2024-03-22T05:06:03.661 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-22T05:06:03.661 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/phlogistonjohn-2024-03-22_03:02:08-orch:cephadm-wip-phlogistonjohn-testing-2024-03-21-1835-distro-default-smithi/7616054 --verbose --description orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/classic} --name phlogistonjohn-2024-03-22_03:02:08-orch:cephadm-wip-phlogistonjohn-testing-2024-03-21-1835-distro-default-smithi --owner scheduled_phlogistonjohn@teuthology -- /home/teuthworker/archive/phlogistonjohn-2024-03-22_03:02:08-orch:cephadm-wip-phlogistonjohn-testing-2024-03-21-1835-distro-default-smithi/7616054/orig.config.yaml 2024-03-22T05:06:03.709 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-22T05:06:03.795 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/phlogistonjohn-2024-03-22_03:02:08-orch:cephadm-wip-phlogistonjohn-testing-2024-03-21-1835-distro-default-smithi/7616054 branch: wip-phlogistonjohn-testing-2024-03-21-1835 description: orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/classic} email: jmulligan@redhat.com first_in_suite: false job_id: '7616054' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: phlogistonjohn-2024-03-22_03:02:08-orch:cephadm-wip-phlogistonjohn-testing-2024-03-21-1835-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-phlogistonjohn-testing-2024-03-21-1835 ceph: conf: global: mon election default strategy: 1 mgr: debug mgr: 20 debug ms: 1 mgr/cephadm/use_agent: false mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: ea01217f8a79678e64ef6a5d68a043a4ef21ca37 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: ea01217f8a79678e64ef6a5d68a043a4ef21ca37 workunit: branch: wip-phlogistonjohn-testing-2024-03-21-1835 sha1: ea01217f8a79678e64ef6a5d68a043a4ef21ca37 owner: scheduled_phlogistonjohn@teuthology priority: 95 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b seed: 5374 sha1: ea01217f8a79678e64ef6a5d68a043a4ef21ca37 sleep_before_teardown: 0 subset: 1/24 suite: orch:cephadm suite_branch: wip-phlogistonjohn-testing-2024-03-21-1835 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_ea01217f8a79678e64ef6a5d68a043a4ef21ca37/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: ea01217f8a79678e64ef6a5d68a043a4ef21ca37 targets: smithi005.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPzVmRysK6cWm/K7TyDn26pLTrKjNjf5YsJF69j9P2iKhMN0Tsq0RxTMyRjh9Q7SWLM4d2lGNxhhCaLJX01mwUE= smithi186.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD/uivWwKGQpDzapTJ7C4+YQ5E/K0dCT4rdP4t5E2UfJKmnfA1EA5gLKYjQDFUneT0wUQfiUXdVbs4MogdIBAFg= tasks: - cephadm: cephadm_branch: v17.2.0 cephadm_git_url: https://github.com/ceph/ceph image: quay.io/ceph/ceph:v17.2.0 - cephadm.shell: env: - sha1 mon.a: - radosgw-admin realm create --rgw-realm=r --default - radosgw-admin zonegroup create --rgw-zonegroup=default --master --default - radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default - radosgw-admin period update --rgw-realm=r --commit - ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000 - ceph osd pool create foo - rbd pool init foo - ceph orch apply iscsi foo u p - sleep 180 - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch ps - ceph versions - ceph -s - ceph orch ls - ceph orch daemon redeploy "mgr.$(ceph mgr dump -f json | jq .standbys | jq .[] | jq -r .name)" --image quay.ceph.io/ceph-ci/ceph:$sha1 - ceph orch ps --refresh - sleep 180 - ceph orch ps - ceph versions - ceph -s - ceph health detail - ceph versions | jq -e '.mgr | length == 2' - ceph mgr fail - sleep 180 - ceph orch daemon redeploy "mgr.$(ceph mgr dump -f json | jq .standbys | jq .[] | jq -r .name)" --image quay.ceph.io/ceph-ci/ceph:$sha1 - ceph orch ps --refresh - sleep 180 - ceph orch ps - ceph versions - ceph health detail - ceph -s - ceph mgr fail - sleep 180 - ceph orch ps - ceph versions - ceph -s - ceph health detail - ceph versions | jq -e '.mgr | length == 1' - ceph mgr fail - sleep 180 - ceph orch ps - ceph orch ls - ceph versions - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph versions | jq -e '.mgr | length == 1' - ceph versions | jq -e '.mgr | keys' | grep $sha1 - ceph versions | jq -e '.overall | length == 2' - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 2' - ceph orch upgrade status - ceph health detail - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mon --hosts $(ceph orch ps | grep mgr.x | awk '{print $2}') - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions | jq -e '.mon | length == 2' - ceph orch upgrade status - ceph health detail - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mon --hosts $(ceph orch ps | grep mgr.y | awk '{print $2}') - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions | jq -e '.mon | length == 1' - ceph versions | jq -e '.mon | keys' | grep $sha1 - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 5' - ceph orch upgrade status - ceph health detail - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types osd --limit 2 - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions | jq -e '.osd | length == 2' - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 7' - ceph orch upgrade status - ceph health detail - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types crash,osd --limit 1 - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions | jq -e '.osd | length == 2' - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 8' - ceph orch upgrade status - ceph health detail - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types crash,osd - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions | jq -e '.osd | length == 1' - ceph versions | jq -e '.osd | keys' | grep $sha1 - ceph orch upgrade status - ceph health detail - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --services rgw.foo - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions | jq -e '.rgw | length == 1' - ceph versions | jq -e '.rgw | keys' | grep $sha1 - ceph orch upgrade status - ceph health detail - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 mon.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph orch upgrade status - ceph health detail - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 - ceph orch ls | grep '^osd ' - cephadm.shell: mon.a: - ceph orch upgrade ls - ceph orch upgrade ls --image quay.io/ceph/ceph --show-all-versions | grep 16.2.0 - ceph orch upgrade ls --image quay.io/ceph/ceph --tags | grep v16.2.2 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-22_03:02:08 tube: smithi user: phlogistonjohn verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-22T05:06:03.795 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_ea01217f8a79678e64ef6a5d68a043a4ef21ca37/qa; will attempt to use it 2024-03-22T05:06:03.796 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_ea01217f8a79678e64ef6a5d68a043a4ef21ca37/qa/tasks 2024-03-22T05:06:03.796 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-22T05:06:03.798 INFO:teuthology.task.internal:Checking packages... 2024-03-22T05:06:03.825 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'ea01217f8a79678e64ef6a5d68a043a4ef21ca37' 2024-03-22T05:06:03.825 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-22T05:06:03.825 INFO:teuthology.packaging:ref: None 2024-03-22T05:06:03.825 INFO:teuthology.packaging:tag: None 2024-03-22T05:06:03.826 INFO:teuthology.packaging:branch: wip-phlogistonjohn-testing-2024-03-21-1835 2024-03-22T05:06:03.826 INFO:teuthology.packaging:sha1: ea01217f8a79678e64ef6a5d68a043a4ef21ca37 2024-03-22T05:06:03.826 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-phlogistonjohn-testing-2024-03-21-1835 2024-03-22T05:06:04.041 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2299.gea01217f 2024-03-22T05:06:04.042 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-22T05:06:04.049 INFO:teuthology.task.internal:no buildpackages task found 2024-03-22T05:06:04.049 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-22T05:06:04.064 INFO:teuthology.task.internal:Saving configuration 2024-03-22T05:06:04.082 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-22T05:06:04.088 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-22T05:06:04.110 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi005.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/phlogistonjohn-2024-03-22_03:02:08-orch:cephadm-wip-phlogistonjohn-testing-2024-03-21-1835-distro-default-smithi/7616054', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-22 05:00:46.649248', 'locked_by': 'scheduled_phlogistonjohn@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPzVmRysK6cWm/K7TyDn26pLTrKjNjf5YsJF69j9P2iKhMN0Tsq0RxTMyRjh9Q7SWLM4d2lGNxhhCaLJX01mwUE='} 2024-03-22T05:06:04.135 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi186.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/phlogistonjohn-2024-03-22_03:02:08-orch:cephadm-wip-phlogistonjohn-testing-2024-03-21-1835-distro-default-smithi/7616054', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-22 05:00:46.647484', 'locked_by': 'scheduled_phlogistonjohn@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD/uivWwKGQpDzapTJ7C4+YQ5E/K0dCT4rdP4t5E2UfJKmnfA1EA5gLKYjQDFUneT0wUQfiUXdVbs4MogdIBAFg='} 2024-03-22T05:06:04.135 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-22T05:06:04.142 INFO:teuthology.task.internal:roles: ubuntu@smithi005.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a'] 2024-03-22T05:06:04.142 INFO:teuthology.task.internal:roles: ubuntu@smithi186.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2024-03-22T05:06:04.142 INFO:teuthology.run_tasks:Running task console_log... 2024-03-22T05:06:04.244 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f4b29ae00d0>, signals=[15]) 2024-03-22T05:06:04.246 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-22T05:06:04.252 INFO:teuthology.task.internal:Opening connections... 2024-03-22T05:06:04.252 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi005.front.sepia.ceph.com 2024-03-22T05:06:04.254 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi005.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-22T05:06:04.361 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi186.front.sepia.ceph.com 2024-03-22T05:06:04.363 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-22T05:06:04.449 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-22T05:06:04.456 DEBUG:teuthology.orchestra.run.smithi005:> uname -m 2024-03-22T05:06:04.473 INFO:teuthology.orchestra.run.smithi005.stdout:x86_64 2024-03-22T05:06:04.473 DEBUG:teuthology.orchestra.run.smithi005:> cat /etc/os-release 2024-03-22T05:06:04.530 INFO:teuthology.orchestra.run.smithi005.stdout:NAME="CentOS Stream" 2024-03-22T05:06:04.530 INFO:teuthology.orchestra.run.smithi005.stdout:VERSION="9" 2024-03-22T05:06:04.530 INFO:teuthology.orchestra.run.smithi005.stdout:ID="centos" 2024-03-22T05:06:04.530 INFO:teuthology.orchestra.run.smithi005.stdout:ID_LIKE="rhel fedora" 2024-03-22T05:06:04.531 INFO:teuthology.orchestra.run.smithi005.stdout:VERSION_ID="9" 2024-03-22T05:06:04.531 INFO:teuthology.orchestra.run.smithi005.stdout:PLATFORM_ID="platform:el9" 2024-03-22T05:06:04.531 INFO:teuthology.orchestra.run.smithi005.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-22T05:06:04.531 INFO:teuthology.orchestra.run.smithi005.stdout:ANSI_COLOR="0;31" 2024-03-22T05:06:04.531 INFO:teuthology.orchestra.run.smithi005.stdout:LOGO="fedora-logo-icon" 2024-03-22T05:06:04.531 INFO:teuthology.orchestra.run.smithi005.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-22T05:06:04.531 INFO:teuthology.orchestra.run.smithi005.stdout:HOME_URL="https://centos.org/" 2024-03-22T05:06:04.531 INFO:teuthology.orchestra.run.smithi005.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-22T05:06:04.531 INFO:teuthology.orchestra.run.smithi005.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-22T05:06:04.531 INFO:teuthology.orchestra.run.smithi005.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-22T05:06:04.532 INFO:teuthology.lock.ops:Updating smithi005.front.sepia.ceph.com on lock server 2024-03-22T05:06:04.552 DEBUG:teuthology.orchestra.run.smithi186:> uname -m 2024-03-22T05:06:04.569 INFO:teuthology.orchestra.run.smithi186.stdout:x86_64 2024-03-22T05:06:04.569 DEBUG:teuthology.orchestra.run.smithi186:> cat /etc/os-release 2024-03-22T05:06:04.625 INFO:teuthology.orchestra.run.smithi186.stdout:NAME="CentOS Stream" 2024-03-22T05:06:04.625 INFO:teuthology.orchestra.run.smithi186.stdout:VERSION="9" 2024-03-22T05:06:04.626 INFO:teuthology.orchestra.run.smithi186.stdout:ID="centos" 2024-03-22T05:06:04.626 INFO:teuthology.orchestra.run.smithi186.stdout:ID_LIKE="rhel fedora" 2024-03-22T05:06:04.626 INFO:teuthology.orchestra.run.smithi186.stdout:VERSION_ID="9" 2024-03-22T05:06:04.626 INFO:teuthology.orchestra.run.smithi186.stdout:PLATFORM_ID="platform:el9" 2024-03-22T05:06:04.626 INFO:teuthology.orchestra.run.smithi186.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-22T05:06:04.626 INFO:teuthology.orchestra.run.smithi186.stdout:ANSI_COLOR="0;31" 2024-03-22T05:06:04.626 INFO:teuthology.orchestra.run.smithi186.stdout:LOGO="fedora-logo-icon" 2024-03-22T05:06:04.626 INFO:teuthology.orchestra.run.smithi186.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-22T05:06:04.626 INFO:teuthology.orchestra.run.smithi186.stdout:HOME_URL="https://centos.org/" 2024-03-22T05:06:04.626 INFO:teuthology.orchestra.run.smithi186.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-22T05:06:04.626 INFO:teuthology.orchestra.run.smithi186.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-22T05:06:04.626 INFO:teuthology.orchestra.run.smithi186.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-22T05:06:04.627 INFO:teuthology.lock.ops:Updating smithi186.front.sepia.ceph.com on lock server 2024-03-22T05:06:04.643 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-22T05:06:04.653 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-22T05:06:04.690 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-22T05:06:04.690 DEBUG:teuthology.orchestra.run.smithi005:> test '!' -e /home/ubuntu/cephtest 2024-03-22T05:06:04.693 DEBUG:teuthology.orchestra.run.smithi186:> test '!' -e /home/ubuntu/cephtest 2024-03-22T05:06:04.709 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-22T05:06:04.715 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-22T05:06:04.715 DEBUG:teuthology.orchestra.run.smithi005:> test -z $(ls -A /var/lib/ceph) 2024-03-22T05:06:04.751 DEBUG:teuthology.orchestra.run.smithi186:> test -z $(ls -A /var/lib/ceph) 2024-03-22T05:06:04.864 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-22T05:06:04.918 INFO:teuthology.run_tasks:Running task kernel... 2024-03-22T05:06:04.933 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-22T05:06:04.933 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-22T05:06:04.934 DEBUG:teuthology.orchestra.run.smithi005:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-22T05:06:04.934 DEBUG:teuthology.orchestra.run.smithi186:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-22T05:06:04.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-22T05:06:04.951 DEBUG:teuthology.orchestra.run.smithi186:> uname -r 2024-03-22T05:06:04.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-22T05:06:04.952 DEBUG:teuthology.orchestra.run.smithi005:> uname -r 2024-03-22T05:06:05.007 INFO:teuthology.orchestra.run.smithi186.stdout:5.14.0-361.el9.x86_64 2024-03-22T05:06:05.007 INFO:teuthology.task.kernel:Running kernel on smithi186: 5.14.0-361.el9.x86_64 2024-03-22T05:06:05.007 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum install -y kernel 2024-03-22T05:06:05.009 INFO:teuthology.orchestra.run.smithi005.stdout:5.14.0-361.el9.x86_64 2024-03-22T05:06:05.009 INFO:teuthology.task.kernel:Running kernel on smithi005: 5.14.0-361.el9.x86_64 2024-03-22T05:06:05.009 DEBUG:teuthology.orchestra.run.smithi005:> sudo yum install -y kernel 2024-03-22T05:06:07.105 INFO:teuthology.orchestra.run.smithi005.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-22T05:06:07.270 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-22T05:06:08.539 INFO:teuthology.orchestra.run.smithi005.stdout:CentOS-9 - BaseOS 7.4 MB/s | 2.0 MB 00:00 2024-03-22T05:06:08.662 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS-9 - BaseOS 6.9 MB/s | 2.0 MB 00:00 2024-03-22T05:06:09.694 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-03-22T05:06:11.587 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - AppStream 22 MB/s | 19 MB 00:00 2024-03-22T05:06:12.387 INFO:teuthology.orchestra.run.smithi005.stdout:CentOS Stream 9 - BaseOS 2.3 MB/s | 8.0 MB 00:03 2024-03-22T05:06:17.038 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - CRB 3.1 MB/s | 6.1 MB 00:01 2024-03-22T05:06:18.505 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - Extras packages 50 kB/s | 16 kB 00:00 2024-03-22T05:06:19.923 INFO:teuthology.orchestra.run.smithi186.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-03-22T05:06:20.796 INFO:teuthology.orchestra.run.smithi005.stdout:CentOS Stream 9 - AppStream 2.6 MB/s | 19 MB 00:07 2024-03-22T05:06:24.906 INFO:teuthology.orchestra.run.smithi186.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-03-22T05:06:25.107 INFO:teuthology.orchestra.run.smithi005.stdout:CentOS Stream 9 - CRB 8.7 MB/s | 6.1 MB 00:00 2024-03-22T05:06:26.647 INFO:teuthology.orchestra.run.smithi005.stdout:CentOS Stream 9 - Extras packages 44 kB/s | 16 kB 00:00 2024-03-22T05:06:26.930 INFO:teuthology.orchestra.run.smithi186.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-22T05:06:26.930 INFO:teuthology.orchestra.run.smithi186.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-22T05:06:27.017 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-03-22T05:06:27.025 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-22T05:06:27.025 INFO:teuthology.orchestra.run.smithi186.stdout: Package Architecture Version Repository Size 2024-03-22T05:06:27.025 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-22T05:06:27.025 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-03-22T05:06:27.025 INFO:teuthology.orchestra.run.smithi186.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-22T05:06:27.025 INFO:teuthology.orchestra.run.smithi186.stdout:Installing dependencies: 2024-03-22T05:06:27.025 INFO:teuthology.orchestra.run.smithi186.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-22T05:06:27.025 INFO:teuthology.orchestra.run.smithi186.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-22T05:06:27.025 INFO:teuthology.orchestra.run.smithi186.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-22T05:06:27.025 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-22T05:06:27.026 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-03-22T05:06:27.026 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2024-03-22T05:06:27.026 INFO:teuthology.orchestra.run.smithi186.stdout:Install 4 Packages 2024-03-22T05:06:27.026 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-22T05:06:27.026 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 76 M 2024-03-22T05:06:27.026 INFO:teuthology.orchestra.run.smithi186.stdout:Installed size: 123 M 2024-03-22T05:06:27.026 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-03-22T05:06:27.499 INFO:teuthology.orchestra.run.smithi186.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 247 kB/s | 44 kB 00:00 2024-03-22T05:06:28.543 INFO:teuthology.orchestra.run.smithi005.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-03-22T05:06:30.316 INFO:teuthology.orchestra.run.smithi186.stdout:(2/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 11 MB/s | 33 MB 00:02 2024-03-22T05:06:31.233 INFO:teuthology.orchestra.run.smithi186.stdout:(3/4): kernel-core-5.14.0-430.el9.x86_64.rpm 3.8 MB/s | 15 MB 00:03 2024-03-22T05:06:31.549 INFO:teuthology.orchestra.run.smithi186.stdout:(4/4): kernel-modules-core-5.14.0-430.el9.x86_6 6.8 MB/s | 27 MB 00:04 2024-03-22T05:06:31.550 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-03-22T05:06:31.550 INFO:teuthology.orchestra.run.smithi186.stdout:Total 17 MB/s | 76 MB 00:04 2024-03-22T05:06:32.219 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-03-22T05:06:32.594 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-03-22T05:06:32.595 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-03-22T05:06:33.580 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-03-22T05:06:33.580 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-03-22T05:06:33.632 INFO:teuthology.orchestra.run.smithi005.stdout:lab-extras 44 kB/s | 1.7 kB 00:00 2024-03-22T05:06:35.822 INFO:teuthology.orchestra.run.smithi005.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-22T05:06:35.823 INFO:teuthology.orchestra.run.smithi005.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-22T05:06:35.915 INFO:teuthology.orchestra.run.smithi005.stdout:Dependencies resolved. 2024-03-22T05:06:35.923 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-03-22T05:06:35.923 INFO:teuthology.orchestra.run.smithi005.stdout: Package Architecture Version Repository Size 2024-03-22T05:06:35.923 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-03-22T05:06:35.924 INFO:teuthology.orchestra.run.smithi005.stdout:Installing: 2024-03-22T05:06:35.924 INFO:teuthology.orchestra.run.smithi005.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-22T05:06:35.924 INFO:teuthology.orchestra.run.smithi005.stdout:Installing dependencies: 2024-03-22T05:06:35.924 INFO:teuthology.orchestra.run.smithi005.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-22T05:06:35.924 INFO:teuthology.orchestra.run.smithi005.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-22T05:06:35.924 INFO:teuthology.orchestra.run.smithi005.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-22T05:06:35.924 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-22T05:06:35.924 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction Summary 2024-03-22T05:06:35.924 INFO:teuthology.orchestra.run.smithi005.stdout:================================================================================ 2024-03-22T05:06:35.924 INFO:teuthology.orchestra.run.smithi005.stdout:Install 4 Packages 2024-03-22T05:06:35.924 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-22T05:06:35.925 INFO:teuthology.orchestra.run.smithi005.stdout:Total download size: 76 M 2024-03-22T05:06:35.925 INFO:teuthology.orchestra.run.smithi005.stdout:Installed size: 123 M 2024-03-22T05:06:35.925 INFO:teuthology.orchestra.run.smithi005.stdout:Downloading Packages: 2024-03-22T05:06:36.047 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-03-22T05:06:36.183 INFO:teuthology.orchestra.run.smithi005.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 240 kB/s | 44 kB 00:00 2024-03-22T05:06:36.636 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-22T05:06:36.732 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-22T05:06:37.183 INFO:teuthology.orchestra.run.smithi005.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 13 MB/s | 15 MB 00:01 2024-03-22T05:06:37.884 INFO:teuthology.orchestra.run.smithi005.stdout:(3/4): kernel-modules-core-5.14.0-430.el9.x86_6 16 MB/s | 27 MB 00:01 2024-03-22T05:06:38.192 INFO:teuthology.orchestra.run.smithi005.stdout:(4/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 15 MB/s | 33 MB 00:02 2024-03-22T05:06:38.192 INFO:teuthology.orchestra.run.smithi005.stdout:-------------------------------------------------------------------------------- 2024-03-22T05:06:38.192 INFO:teuthology.orchestra.run.smithi005.stdout:Total 33 MB/s | 76 MB 00:02 2024-03-22T05:06:38.888 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction check 2024-03-22T05:06:39.303 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction check succeeded. 2024-03-22T05:06:39.304 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction test 2024-03-22T05:06:39.488 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-22T05:06:39.648 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-22T05:06:40.383 INFO:teuthology.orchestra.run.smithi005.stdout:Transaction test succeeded. 2024-03-22T05:06:40.384 INFO:teuthology.orchestra.run.smithi005.stdout:Running transaction 2024-03-22T05:06:43.007 INFO:teuthology.orchestra.run.smithi005.stdout: Preparing : 1/1 2024-03-22T05:06:43.631 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-22T05:06:43.725 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-22T05:06:44.342 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-22T05:06:44.435 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-22T05:06:46.627 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-22T05:06:46.760 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-22T05:06:49.184 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-22T05:06:51.839 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-22T05:06:51.885 INFO:teuthology.orchestra.run.smithi005.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-22T05:06:56.991 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-22T05:07:18.292 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-22T05:07:18.292 INFO:teuthology.orchestra.run.smithi186.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-22T05:07:18.292 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-22T05:07:18.337 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-22T05:07:18.699 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-22T05:07:18.699 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-22T05:07:18.699 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-22T05:07:18.699 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-22T05:07:19.095 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-22T05:07:19.095 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-22T05:07:19.096 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-03-22T05:07:19.096 INFO:teuthology.orchestra.run.smithi186.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-22T05:07:19.096 INFO:teuthology.orchestra.run.smithi186.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-22T05:07:19.096 INFO:teuthology.orchestra.run.smithi186.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-22T05:07:19.096 INFO:teuthology.orchestra.run.smithi186.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-22T05:07:19.096 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-22T05:07:19.096 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-03-22T05:07:19.427 DEBUG:teuthology.orchestra.run.smithi186:> rpm -q kernel | sort -rV | head -n 1 2024-03-22T05:07:19.521 INFO:teuthology.orchestra.run.smithi186.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-22T05:07:19.521 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-22T05:07:19.521 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-22T05:07:19.521 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-22T05:07:19.521 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-22T05:07:19.522 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-22T05:07:19.522 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi186.front.sepia.ceph.com, path=None, version=distro) 2024-03-22T05:07:19.522 DEBUG:teuthology.orchestra.run.smithi186:> rpm -q kernel | sort -rV | head -n 1 2024-03-22T05:07:19.600 INFO:teuthology.orchestra.run.smithi186.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-22T05:07:19.601 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-22T05:07:19.602 DEBUG:teuthology.orchestra.run.smithi186:> sudo rpm -qi grub2-tools 2024-03-22T05:07:19.644 INFO:teuthology.orchestra.run.smithi186.stdout:Name : grub2-tools 2024-03-22T05:07:19.644 INFO:teuthology.orchestra.run.smithi186.stdout:Epoch : 1 2024-03-22T05:07:19.644 INFO:teuthology.orchestra.run.smithi186.stdout:Version : 2.06 2024-03-22T05:07:19.644 INFO:teuthology.orchestra.run.smithi186.stdout:Release : 68.el9 2024-03-22T05:07:19.644 INFO:teuthology.orchestra.run.smithi186.stdout:Architecture: x86_64 2024-03-22T05:07:19.644 INFO:teuthology.orchestra.run.smithi186.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-22T05:07:19.644 INFO:teuthology.orchestra.run.smithi186.stdout:Group : Unspecified 2024-03-22T05:07:19.644 INFO:teuthology.orchestra.run.smithi186.stdout:Size : 8264254 2024-03-22T05:07:19.645 INFO:teuthology.orchestra.run.smithi186.stdout:License : GPLv3+ 2024-03-22T05:07:19.645 INFO:teuthology.orchestra.run.smithi186.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-22T05:07:19.645 INFO:teuthology.orchestra.run.smithi186.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-22T05:07:19.645 INFO:teuthology.orchestra.run.smithi186.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-22T05:07:19.645 INFO:teuthology.orchestra.run.smithi186.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-22T05:07:19.645 INFO:teuthology.orchestra.run.smithi186.stdout:Packager : builder@centos.org 2024-03-22T05:07:19.645 INFO:teuthology.orchestra.run.smithi186.stdout:Vendor : CentOS 2024-03-22T05:07:19.645 INFO:teuthology.orchestra.run.smithi186.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-22T05:07:19.645 INFO:teuthology.orchestra.run.smithi186.stdout:Summary : Support tools for GRUB. 2024-03-22T05:07:19.645 INFO:teuthology.orchestra.run.smithi186.stdout:Description : 2024-03-22T05:07:19.645 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-22T05:07:19.645 INFO:teuthology.orchestra.run.smithi186.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-22T05:07:19.645 INFO:teuthology.orchestra.run.smithi186.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-22T05:07:19.645 INFO:teuthology.orchestra.run.smithi186.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-22T05:07:19.645 INFO:teuthology.orchestra.run.smithi186.stdout:hardware devices. 2024-03-22T05:07:19.646 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-22T05:07:19.646 INFO:teuthology.orchestra.run.smithi186.stdout:This subpackage provides tools for support of all platforms. 2024-03-22T05:07:19.646 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-22T05:07:19.647 INFO:teuthology.task.kernel:Updating grub on smithi186 to boot 5.14.0-430.el9.x86_64 2024-03-22T05:07:19.647 DEBUG:teuthology.orchestra.run.smithi186:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-22T05:07:20.271 INFO:teuthology.orchestra.run.smithi186.stderr:Generating grub configuration file ... 2024-03-22T05:07:21.620 INFO:teuthology.orchestra.run.smithi186.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-22T05:07:21.651 INFO:teuthology.orchestra.run.smithi186.stderr:done 2024-03-22T05:07:21.653 DEBUG:teuthology.orchestra.run.smithi186:> mktemp 2024-03-22T05:07:21.667 INFO:teuthology.orchestra.run.smithi186.stdout:/tmp/tmp.RyD6yriJmN 2024-03-22T05:07:21.668 DEBUG:teuthology.orchestra.run.smithi186:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.RyD6yriJmN 2024-03-22T05:07:21.736 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod 0666 /tmp/tmp.RyD6yriJmN 2024-03-22T05:07:21.879 DEBUG:teuthology.orchestra.remote:smithi186:/tmp/tmp.RyD6yriJmN is 6KB 2024-03-22T05:07:21.927 DEBUG:teuthology.orchestra.run.smithi186:> rm -fr /tmp/tmp.RyD6yriJmN 2024-03-22T05:07:21.941 DEBUG:teuthology.orchestra.run.smithi186:> sudo /bin/ls /boot/loader/entries || true 2024-03-22T05:07:22.014 INFO:teuthology.orchestra.run.smithi186.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-22T05:07:22.014 INFO:teuthology.orchestra.run.smithi186.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-22T05:07:22.014 INFO:teuthology.orchestra.run.smithi186.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-22T05:07:22.014 INFO:teuthology.orchestra.run.smithi186.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-22T05:07:22.016 DEBUG:teuthology.orchestra.run.smithi186:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-22T05:07:22.181 DEBUG:teuthology.orchestra.run.smithi186:> sudo shutdown -r now 2024-03-22T05:07:27.687 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-22T05:07:27.688 INFO:teuthology.orchestra.run.smithi005.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-22T05:07:27.688 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-22T05:07:27.733 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-22T05:07:28.135 INFO:teuthology.orchestra.run.smithi005.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-22T05:07:28.135 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-22T05:07:28.135 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-22T05:07:28.135 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-22T05:07:28.560 INFO:teuthology.orchestra.run.smithi005.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-22T05:07:28.560 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-22T05:07:28.560 INFO:teuthology.orchestra.run.smithi005.stdout:Installed: 2024-03-22T05:07:28.560 INFO:teuthology.orchestra.run.smithi005.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-22T05:07:28.560 INFO:teuthology.orchestra.run.smithi005.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-22T05:07:28.560 INFO:teuthology.orchestra.run.smithi005.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-22T05:07:28.560 INFO:teuthology.orchestra.run.smithi005.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-22T05:07:28.560 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-22T05:07:28.560 INFO:teuthology.orchestra.run.smithi005.stdout:Complete! 2024-03-22T05:07:28.909 DEBUG:teuthology.orchestra.run.smithi005:> rpm -q kernel | sort -rV | head -n 1 2024-03-22T05:07:29.012 INFO:teuthology.orchestra.run.smithi005.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-22T05:07:29.013 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-22T05:07:29.013 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-22T05:07:29.014 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-22T05:07:29.014 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-22T05:07:29.014 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-22T05:07:29.014 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi005.front.sepia.ceph.com, path=None, version=distro) 2024-03-22T05:07:29.014 DEBUG:teuthology.orchestra.run.smithi005:> rpm -q kernel | sort -rV | head -n 1 2024-03-22T05:07:29.097 INFO:teuthology.orchestra.run.smithi005.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-22T05:07:29.097 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-22T05:07:29.098 DEBUG:teuthology.orchestra.run.smithi005:> sudo rpm -qi grub2-tools 2024-03-22T05:07:29.135 INFO:teuthology.orchestra.run.smithi005.stdout:Name : grub2-tools 2024-03-22T05:07:29.135 INFO:teuthology.orchestra.run.smithi005.stdout:Epoch : 1 2024-03-22T05:07:29.135 INFO:teuthology.orchestra.run.smithi005.stdout:Version : 2.06 2024-03-22T05:07:29.135 INFO:teuthology.orchestra.run.smithi005.stdout:Release : 68.el9 2024-03-22T05:07:29.135 INFO:teuthology.orchestra.run.smithi005.stdout:Architecture: x86_64 2024-03-22T05:07:29.135 INFO:teuthology.orchestra.run.smithi005.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-22T05:07:29.135 INFO:teuthology.orchestra.run.smithi005.stdout:Group : Unspecified 2024-03-22T05:07:29.136 INFO:teuthology.orchestra.run.smithi005.stdout:Size : 8264254 2024-03-22T05:07:29.136 INFO:teuthology.orchestra.run.smithi005.stdout:License : GPLv3+ 2024-03-22T05:07:29.136 INFO:teuthology.orchestra.run.smithi005.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-22T05:07:29.136 INFO:teuthology.orchestra.run.smithi005.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-22T05:07:29.136 INFO:teuthology.orchestra.run.smithi005.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-22T05:07:29.136 INFO:teuthology.orchestra.run.smithi005.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-22T05:07:29.136 INFO:teuthology.orchestra.run.smithi005.stdout:Packager : builder@centos.org 2024-03-22T05:07:29.136 INFO:teuthology.orchestra.run.smithi005.stdout:Vendor : CentOS 2024-03-22T05:07:29.136 INFO:teuthology.orchestra.run.smithi005.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-22T05:07:29.136 INFO:teuthology.orchestra.run.smithi005.stdout:Summary : Support tools for GRUB. 2024-03-22T05:07:29.136 INFO:teuthology.orchestra.run.smithi005.stdout:Description : 2024-03-22T05:07:29.136 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-22T05:07:29.136 INFO:teuthology.orchestra.run.smithi005.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-22T05:07:29.136 INFO:teuthology.orchestra.run.smithi005.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-22T05:07:29.136 INFO:teuthology.orchestra.run.smithi005.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-22T05:07:29.137 INFO:teuthology.orchestra.run.smithi005.stdout:hardware devices. 2024-03-22T05:07:29.137 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-22T05:07:29.137 INFO:teuthology.orchestra.run.smithi005.stdout:This subpackage provides tools for support of all platforms. 2024-03-22T05:07:29.137 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-22T05:07:29.137 INFO:teuthology.task.kernel:Updating grub on smithi005 to boot 5.14.0-430.el9.x86_64 2024-03-22T05:07:29.138 DEBUG:teuthology.orchestra.run.smithi005:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-22T05:07:29.752 INFO:teuthology.orchestra.run.smithi005.stderr:Generating grub configuration file ... 2024-03-22T05:07:31.128 INFO:teuthology.orchestra.run.smithi005.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-22T05:07:31.161 INFO:teuthology.orchestra.run.smithi005.stderr:done 2024-03-22T05:07:31.163 DEBUG:teuthology.orchestra.run.smithi005:> mktemp 2024-03-22T05:07:31.179 INFO:teuthology.orchestra.run.smithi005.stdout:/tmp/tmp.Bn5h7HBDgo 2024-03-22T05:07:31.179 DEBUG:teuthology.orchestra.run.smithi005:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Bn5h7HBDgo 2024-03-22T05:07:31.268 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod 0666 /tmp/tmp.Bn5h7HBDgo 2024-03-22T05:07:31.373 DEBUG:teuthology.orchestra.remote:smithi005:/tmp/tmp.Bn5h7HBDgo is 6KB 2024-03-22T05:07:31.392 DEBUG:teuthology.orchestra.run.smithi005:> rm -fr /tmp/tmp.Bn5h7HBDgo 2024-03-22T05:07:31.406 DEBUG:teuthology.orchestra.run.smithi005:> sudo /bin/ls /boot/loader/entries || true 2024-03-22T05:07:31.481 INFO:teuthology.orchestra.run.smithi005.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-22T05:07:31.481 INFO:teuthology.orchestra.run.smithi005.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-22T05:07:31.482 INFO:teuthology.orchestra.run.smithi005.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-22T05:07:31.482 INFO:teuthology.orchestra.run.smithi005.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-22T05:07:31.483 DEBUG:teuthology.orchestra.run.smithi005:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-22T05:07:31.633 DEBUG:teuthology.orchestra.run.smithi005:> sudo shutdown -r now 2024-03-22T05:07:52.195 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-22T05:07:52.196 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-03-22T05:07:52.197 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-22T05:08:01.642 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-22T05:08:01.643 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi005.front.sepia.ceph.com' 2024-03-22T05:08:01.643 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi005.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-22T05:08:20.126 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.5 2024-03-22T05:08:26.782 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-03-22T05:08:29.129 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi005.front.sepia.ceph.com' 2024-03-22T05:08:29.129 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi005.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-22T05:08:32.190 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.5 2024-03-22T05:08:35.783 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-03-22T05:08:35.784 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-22T05:08:44.199 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi005.front.sepia.ceph.com' 2024-03-22T05:08:44.200 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi005.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-22T05:08:54.174 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-03-22T05:09:06.183 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-03-22T05:09:06.184 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-22T05:09:09.246 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.186 2024-03-22T05:09:16.063 DEBUG:teuthology.orchestra.run.smithi005:> true 2024-03-22T05:09:16.444 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi005.front.sepia.ceph.com' 2024-03-22T05:09:16.444 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-430.el9.x86_64"... 2024-03-22T05:09:16.444 DEBUG:teuthology.orchestra.run.smithi005:> uname -r 2024-03-22T05:09:16.460 INFO:teuthology.orchestra.run.smithi005.stdout:5.14.0-430.el9.x86_64 2024-03-22T05:09:16.461 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-22T05:09:16.461 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-22T05:09:16.461 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-22T05:09:17.462 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-22T05:09:17.462 DEBUG:teuthology.orchestra.run.smithi005:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-22T05:09:17.562 INFO:teuthology.orchestra.run.smithi005.stdout:ttyS1 2024-03-22T05:09:17.589 DEBUG:teuthology.parallel:result is None 2024-03-22T05:09:24.247 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-03-22T05:09:24.248 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-22T05:09:24.632 DEBUG:teuthology.orchestra.run.smithi186:> true 2024-03-22T05:09:25.028 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-03-22T05:09:25.028 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-430.el9.x86_64"... 2024-03-22T05:09:25.028 DEBUG:teuthology.orchestra.run.smithi186:> uname -r 2024-03-22T05:09:25.044 INFO:teuthology.orchestra.run.smithi186.stdout:5.14.0-430.el9.x86_64 2024-03-22T05:09:25.044 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-22T05:09:25.044 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-22T05:09:25.044 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-22T05:09:26.045 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-22T05:09:26.045 DEBUG:teuthology.orchestra.run.smithi186:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-22T05:09:26.109 INFO:teuthology.orchestra.run.smithi186.stdout:ttyS1 2024-03-22T05:09:26.143 DEBUG:teuthology.parallel:result is None 2024-03-22T05:09:26.144 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-22T05:09:26.152 INFO:teuthology.task.internal:Creating test directory... 2024-03-22T05:09:26.152 DEBUG:teuthology.orchestra.run.smithi005:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-22T05:09:26.154 DEBUG:teuthology.orchestra.run.smithi186:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-22T05:09:26.172 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-22T05:09:26.190 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-22T05:09:26.197 INFO:teuthology.task.internal:Creating archive directory... 2024-03-22T05:09:26.197 DEBUG:teuthology.orchestra.run.smithi005:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-22T05:09:26.212 DEBUG:teuthology.orchestra.run.smithi186:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-22T05:09:26.254 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-22T05:09:26.261 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-22T05:09:26.261 DEBUG:teuthology.orchestra.run.smithi005:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-22T05:09:26.280 DEBUG:teuthology.orchestra.run.smithi186:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-22T05:09:26.316 INFO:teuthology.orchestra.run.smithi005.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-22T05:09:26.331 INFO:teuthology.orchestra.run.smithi005.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-22T05:09:26.336 INFO:teuthology.orchestra.run.smithi186.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-22T05:09:26.351 INFO:teuthology.orchestra.run.smithi186.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-22T05:09:26.353 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-22T05:09:26.363 INFO:teuthology.task.internal:Configuring sudo... 2024-03-22T05:09:26.363 DEBUG:teuthology.orchestra.run.smithi005:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-22T05:09:26.375 DEBUG:teuthology.orchestra.run.smithi186:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-22T05:09:26.404 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-22T05:09:26.414 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-22T05:09:26.415 DEBUG:teuthology.orchestra.run.smithi005:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-22T05:09:26.445 DEBUG:teuthology.orchestra.run.smithi186:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-22T05:09:26.461 DEBUG:teuthology.orchestra.run.smithi005:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-22T05:09:26.540 DEBUG:teuthology.orchestra.run.smithi005:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-22T05:09:26.595 DEBUG:teuthology.orchestra.run.smithi005:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-22T05:09:26.677 DEBUG:teuthology.orchestra.run.smithi005:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-22T05:09:26.720 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-03-22T05:09:26.720 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-22T05:09:26.787 DEBUG:teuthology.orchestra.run.smithi186:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-22T05:09:26.831 DEBUG:teuthology.orchestra.run.smithi186:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-22T05:09:26.887 DEBUG:teuthology.orchestra.run.smithi186:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-22T05:09:26.969 DEBUG:teuthology.orchestra.run.smithi186:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-22T05:09:27.014 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-03-22T05:09:27.014 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-22T05:09:27.082 DEBUG:teuthology.orchestra.run.smithi005:> sudo service rsyslog restart 2024-03-22T05:09:27.085 DEBUG:teuthology.orchestra.run.smithi186:> sudo service rsyslog restart 2024-03-22T05:09:27.130 INFO:teuthology.orchestra.run.smithi005.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-22T05:09:27.171 INFO:teuthology.orchestra.run.smithi186.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-22T05:09:27.684 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-22T05:09:27.691 INFO:teuthology.task.internal:Starting timer... 2024-03-22T05:09:27.692 INFO:teuthology.run_tasks:Running task pcp... 2024-03-22T05:09:27.703 INFO:teuthology.run_tasks:Running task selinux... 2024-03-22T05:09:27.714 DEBUG:teuthology.orchestra.run.smithi005:> sudo service auditd rotate 2024-03-22T05:09:27.760 INFO:teuthology.orchestra.run.smithi005.stdout:Rotating logs: 2024-03-22T05:09:27.762 DEBUG:teuthology.orchestra.run.smithi186:> sudo service auditd rotate 2024-03-22T05:09:27.804 INFO:teuthology.orchestra.run.smithi186.stdout:Rotating logs: 2024-03-22T05:09:27.806 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-22T05:09:27.806 DEBUG:teuthology.orchestra.run.smithi005:> /usr/sbin/getenforce 2024-03-22T05:09:27.839 INFO:teuthology.orchestra.run.smithi005.stdout:Permissive 2024-03-22T05:09:27.840 DEBUG:teuthology.orchestra.run.smithi186:> /usr/sbin/getenforce 2024-03-22T05:09:27.898 INFO:teuthology.orchestra.run.smithi186.stdout:Permissive 2024-03-22T05:09:27.898 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi005.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi186.front.sepia.ceph.com': 'permissive'} 2024-03-22T05:09:27.899 DEBUG:teuthology.orchestra.run.smithi005:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-22T05:09:27.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-22T05:09:27.928 DEBUG:teuthology.orchestra.run.smithi186:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-22T05:09:27.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-22T05:09:27.956 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-22T05:09:27.956 DEBUG:teuthology.orchestra.run.smithi005:> sudo /usr/sbin/setenforce permissive 2024-03-22T05:09:28.008 DEBUG:teuthology.orchestra.run.smithi186:> sudo /usr/sbin/setenforce permissive 2024-03-22T05:09:28.042 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-22T05:09:28.053 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-22T05:09:28.070 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-22T05:09:28.072 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi005.front.sepia.ceph.com,smithi186.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-22T05:13:45.654 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi005.front.sepia.ceph.com'), Remote(name='ubuntu@smithi186.front.sepia.ceph.com')] 2024-03-22T05:13:45.655 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi005.front.sepia.ceph.com' 2024-03-22T05:13:45.656 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi005.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-22T05:13:45.746 DEBUG:teuthology.orchestra.run.smithi005:> true 2024-03-22T05:13:45.821 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi005.front.sepia.ceph.com' 2024-03-22T05:13:45.821 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-03-22T05:13:45.822 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-22T05:13:45.910 DEBUG:teuthology.orchestra.run.smithi186:> true 2024-03-22T05:13:45.995 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-03-22T05:13:45.995 INFO:teuthology.run_tasks:Running task clock... 2024-03-22T05:13:46.005 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-22T05:13:46.006 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-22T05:13:46.006 DEBUG:teuthology.orchestra.run.smithi005:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-22T05:13:46.009 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-22T05:13:46.009 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-22T05:13:46.038 INFO:teuthology.orchestra.run.smithi005.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-22T05:13:46.054 INFO:teuthology.orchestra.run.smithi005.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-22T05:13:46.069 INFO:teuthology.orchestra.run.smithi186.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-22T05:13:46.083 INFO:teuthology.orchestra.run.smithi005.stderr:sudo: ntpd: command not found 2024-03-22T05:13:46.084 INFO:teuthology.orchestra.run.smithi186.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-22T05:13:46.097 INFO:teuthology.orchestra.run.smithi005.stdout:506 Cannot talk to daemon 2024-03-22T05:13:46.113 INFO:teuthology.orchestra.run.smithi005.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-22T05:13:46.114 INFO:teuthology.orchestra.run.smithi186.stderr:sudo: ntpd: command not found 2024-03-22T05:13:46.128 INFO:teuthology.orchestra.run.smithi186.stdout:506 Cannot talk to daemon 2024-03-22T05:13:46.129 INFO:teuthology.orchestra.run.smithi005.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-22T05:13:46.145 INFO:teuthology.orchestra.run.smithi186.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-22T05:13:46.161 INFO:teuthology.orchestra.run.smithi186.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-22T05:13:46.183 INFO:teuthology.orchestra.run.smithi005.stderr:bash: line 1: ntpq: command not found 2024-03-22T05:13:46.186 INFO:teuthology.orchestra.run.smithi005.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-22T05:13:46.186 INFO:teuthology.orchestra.run.smithi005.stdout:=============================================================================== 2024-03-22T05:13:46.186 INFO:teuthology.orchestra.run.smithi005.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-22T05:13:46.186 INFO:teuthology.orchestra.run.smithi005.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-22T05:13:46.186 INFO:teuthology.orchestra.run.smithi005.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-22T05:13:46.186 INFO:teuthology.orchestra.run.smithi005.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-22T05:13:46.213 INFO:teuthology.orchestra.run.smithi186.stderr:bash: line 1: ntpq: command not found 2024-03-22T05:13:46.216 INFO:teuthology.orchestra.run.smithi186.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-22T05:13:46.216 INFO:teuthology.orchestra.run.smithi186.stdout:=============================================================================== 2024-03-22T05:13:46.216 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-22T05:13:46.217 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-22T05:13:46.217 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-22T05:13:46.217 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-22T05:13:46.217 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-22T05:13:46.322 INFO:tasks.cephadm:Config: {'cephadm_branch': 'v17.2.0', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'quay.io/ceph/ceph:v17.2.0', 'conf': {'global': {'mon election default strategy': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr/cephadm/use_agent': False}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'ea01217f8a79678e64ef6a5d68a043a4ef21ca37'} 2024-03-22T05:13:46.322 INFO:tasks.cephadm:Cluster image is quay.io/ceph/ceph:v17.2.0 2024-03-22T05:13:46.323 INFO:tasks.cephadm:Cluster fsid is f59f36e2-e80a-11ee-95cd-87774f69a715 2024-03-22T05:13:46.323 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-22T05:13:46.323 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.5', 'mon.c': '[v2:172.21.15.5:3301,v1:172.21.15.5:6790]', 'mon.b': '172.21.15.186'} 2024-03-22T05:13:46.323 INFO:tasks.cephadm:First mon is mon.a on smithi005 2024-03-22T05:13:46.324 INFO:tasks.cephadm:First mgr is y 2024-03-22T05:13:46.324 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-22T05:13:46.324 DEBUG:teuthology.orchestra.run.smithi005:> sudo hostname $(hostname -s) 2024-03-22T05:13:46.352 DEBUG:teuthology.orchestra.run.smithi186:> sudo hostname $(hostname -s) 2024-03-22T05:13:46.380 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v17.2.0)... 2024-03-22T05:13:46.380 DEBUG:teuthology.orchestra.run.smithi005:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-22T05:13:46.620 INFO:teuthology.orchestra.run.smithi005.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Mar 22 05:13 /home/ubuntu/cephtest/cephadm 2024-03-22T05:13:46.620 DEBUG:teuthology.orchestra.run.smithi186:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-22T05:13:46.790 INFO:teuthology.orchestra.run.smithi186.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Mar 22 05:13 /home/ubuntu/cephtest/cephadm 2024-03-22T05:13:46.790 DEBUG:teuthology.orchestra.run.smithi005:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-22T05:13:46.805 DEBUG:teuthology.orchestra.run.smithi186:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-22T05:13:46.831 INFO:tasks.cephadm:Pulling image quay.io/ceph/ceph:v17.2.0 on all hosts... 2024-03-22T05:13:46.832 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-03-22T05:13:46.848 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-03-22T05:13:47.705 INFO:teuthology.orchestra.run.smithi005.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-03-22T05:13:47.744 INFO:teuthology.orchestra.run.smithi186.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-03-22T05:14:08.121 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-03-22T05:14:08.121 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-03-22T05:14:08.122 INFO:teuthology.orchestra.run.smithi005.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-03-22T05:14:08.122 INFO:teuthology.orchestra.run.smithi005.stdout: "repo_digests": [ 2024-03-22T05:14:08.122 INFO:teuthology.orchestra.run.smithi005.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-03-22T05:14:08.122 INFO:teuthology.orchestra.run.smithi005.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-03-22T05:14:08.122 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-03-22T05:14:08.122 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-03-22T05:14:09.909 INFO:teuthology.orchestra.run.smithi186.stdout:{ 2024-03-22T05:14:09.909 INFO:teuthology.orchestra.run.smithi186.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-03-22T05:14:09.909 INFO:teuthology.orchestra.run.smithi186.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-03-22T05:14:09.910 INFO:teuthology.orchestra.run.smithi186.stdout: "repo_digests": [ 2024-03-22T05:14:09.910 INFO:teuthology.orchestra.run.smithi186.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-03-22T05:14:09.910 INFO:teuthology.orchestra.run.smithi186.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-03-22T05:14:09.910 INFO:teuthology.orchestra.run.smithi186.stdout: ] 2024-03-22T05:14:09.910 INFO:teuthology.orchestra.run.smithi186.stdout:} 2024-03-22T05:14:09.928 DEBUG:teuthology.orchestra.run.smithi005:> sudo mkdir -p /etc/ceph 2024-03-22T05:14:09.964 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /etc/ceph 2024-03-22T05:14:09.994 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod 777 /etc/ceph 2024-03-22T05:14:10.033 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod 777 /etc/ceph 2024-03-22T05:14:10.061 INFO:tasks.cephadm:Writing seed config... 2024-03-22T05:14:10.062 INFO:tasks.cephadm: override: [global] mon election default strategy = 1 2024-03-22T05:14:10.062 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-22T05:14:10.062 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-22T05:14:10.062 INFO:tasks.cephadm: override: [mgr] mgr/cephadm/use_agent = False 2024-03-22T05:14:10.062 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-22T05:14:10.062 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-22T05:14:10.063 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-22T05:14:10.063 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-22T05:14:10.063 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-22T05:14:10.063 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-03-22T05:14:10.063 DEBUG:teuthology.orchestra.run.smithi005:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-22T05:14:10.090 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = f59f36e2-e80a-11ee-95cd-87774f69a715 mon election default strategy = 1 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 mgr/cephadm/use_agent = False [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-03-22T05:14:10.090 DEBUG:teuthology.orchestra.run.smithi005:mon.a> sudo journalctl -f -n 0 -u ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@mon.a.service 2024-03-22T05:14:10.133 DEBUG:teuthology.orchestra.run.smithi005:mgr.y> sudo journalctl -f -n 0 -u ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@mgr.y.service 2024-03-22T05:14:10.175 INFO:tasks.cephadm:Bootstrapping... 2024-03-22T05:14:10.176 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 -v bootstrap --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.5 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-22T05:14:10.378 INFO:teuthology.orchestra.run.smithi005.stderr:-------------------------------------------------------------------------------- 2024-03-22T05:14:10.378 INFO:teuthology.orchestra.run.smithi005.stderr:cephadm ['--image', 'quay.io/ceph/ceph:v17.2.0', '-v', 'bootstrap', '--fsid', 'f59f36e2-e80a-11ee-95cd-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.5', '--skip-admin-label'] 2024-03-22T05:14:10.401 INFO:teuthology.orchestra.run.smithi005.stderr:/bin/podman: 4.9.3-dev 2024-03-22T05:14:10.403 INFO:teuthology.orchestra.run.smithi005.stderr:Verifying podman|docker is present... 2024-03-22T05:14:10.424 INFO:teuthology.orchestra.run.smithi005.stderr:/bin/podman: 4.9.3-dev 2024-03-22T05:14:10.427 INFO:teuthology.orchestra.run.smithi005.stderr:Verifying lvm2 is present... 2024-03-22T05:14:10.427 INFO:teuthology.orchestra.run.smithi005.stderr:Verifying time synchronization is in place... 2024-03-22T05:14:10.433 INFO:teuthology.orchestra.run.smithi005.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-03-22T05:14:10.439 INFO:teuthology.orchestra.run.smithi005.stderr:systemctl: inactive 2024-03-22T05:14:10.446 INFO:teuthology.orchestra.run.smithi005.stderr:systemctl: enabled 2024-03-22T05:14:10.452 INFO:teuthology.orchestra.run.smithi005.stderr:systemctl: active 2024-03-22T05:14:10.452 INFO:teuthology.orchestra.run.smithi005.stderr:Unit chronyd.service is enabled and running 2024-03-22T05:14:10.452 INFO:teuthology.orchestra.run.smithi005.stderr:Repeating the final host check... 2024-03-22T05:14:10.472 INFO:teuthology.orchestra.run.smithi005.stderr:/bin/podman: 4.9.3-dev 2024-03-22T05:14:10.474 INFO:teuthology.orchestra.run.smithi005.stderr:podman (/bin/podman) version 4.9.3 is present 2024-03-22T05:14:10.474 INFO:teuthology.orchestra.run.smithi005.stderr:systemctl is present 2024-03-22T05:14:10.475 INFO:teuthology.orchestra.run.smithi005.stderr:lvcreate is present 2024-03-22T05:14:10.480 INFO:teuthology.orchestra.run.smithi005.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-03-22T05:14:10.486 INFO:teuthology.orchestra.run.smithi005.stderr:systemctl: inactive 2024-03-22T05:14:10.493 INFO:teuthology.orchestra.run.smithi005.stderr:systemctl: enabled 2024-03-22T05:14:10.499 INFO:teuthology.orchestra.run.smithi005.stderr:systemctl: active 2024-03-22T05:14:10.500 INFO:teuthology.orchestra.run.smithi005.stderr:Unit chronyd.service is enabled and running 2024-03-22T05:14:10.500 INFO:teuthology.orchestra.run.smithi005.stderr:Host looks OK 2024-03-22T05:14:10.500 INFO:teuthology.orchestra.run.smithi005.stderr:Cluster fsid: f59f36e2-e80a-11ee-95cd-87774f69a715 2024-03-22T05:14:10.500 INFO:teuthology.orchestra.run.smithi005.stderr:Acquiring lock 140531704188832 on /run/cephadm/f59f36e2-e80a-11ee-95cd-87774f69a715.lock 2024-03-22T05:14:10.500 INFO:teuthology.orchestra.run.smithi005.stderr:Lock 140531704188832 acquired on /run/cephadm/f59f36e2-e80a-11ee-95cd-87774f69a715.lock 2024-03-22T05:14:10.501 INFO:teuthology.orchestra.run.smithi005.stderr:Verifying IP 172.21.15.5 port 3300 ... 2024-03-22T05:14:10.501 INFO:teuthology.orchestra.run.smithi005.stderr:Verifying IP 172.21.15.5 port 6789 ... 2024-03-22T05:14:10.501 INFO:teuthology.orchestra.run.smithi005.stderr:Base mon IP is 172.21.15.5, final addrv is [v2:172.21.15.5:3300,v1:172.21.15.5:6789] 2024-03-22T05:14:10.503 INFO:teuthology.orchestra.run.smithi005.stderr:/sbin/ip: default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.5 metric 100 2024-03-22T05:14:10.503 INFO:teuthology.orchestra.run.smithi005.stderr:/sbin/ip: 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.5 metric 100 2024-03-22T05:14:10.506 INFO:teuthology.orchestra.run.smithi005.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-03-22T05:14:10.506 INFO:teuthology.orchestra.run.smithi005.stderr:/sbin/ip: fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-03-22T05:14:10.506 INFO:teuthology.orchestra.run.smithi005.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-03-22T05:14:10.508 INFO:teuthology.orchestra.run.smithi005.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-22T05:14:10.508 INFO:teuthology.orchestra.run.smithi005.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-03-22T05:14:10.508 INFO:teuthology.orchestra.run.smithi005.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-03-22T05:14:10.508 INFO:teuthology.orchestra.run.smithi005.stderr:/sbin/ip: 4: ens1f0: mtu 1500 state UP qlen 1000 2024-03-22T05:14:10.508 INFO:teuthology.orchestra.run.smithi005.stderr:/sbin/ip: inet6 fe80::ec4:7aff:febd:1442/64 scope link noprefixroute 2024-03-22T05:14:10.508 INFO:teuthology.orchestra.run.smithi005.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-03-22T05:14:10.509 INFO:teuthology.orchestra.run.smithi005.stderr:Mon IP `172.21.15.5` is in CIDR network `172.21.0.0/20` 2024-03-22T05:14:10.509 INFO:teuthology.orchestra.run.smithi005.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-22T05:14:10.510 INFO:teuthology.orchestra.run.smithi005.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-03-22T05:14:10.613 INFO:teuthology.orchestra.run.smithi005.stderr:/bin/podman: Trying to pull quay.io/ceph/ceph:v17.2.0... 2024-03-22T05:14:11.009 INFO:teuthology.orchestra.run.smithi005.stderr:/bin/podman: Getting image source signatures 2024-03-22T05:14:11.010 INFO:teuthology.orchestra.run.smithi005.stderr:/bin/podman: Copying blob sha256:33ca8fff7868c4dc0c11e09bca97c720eb9cfbab7221216754367dd8de70388a 2024-03-22T05:14:11.010 INFO:teuthology.orchestra.run.smithi005.stderr:/bin/podman: Copying blob sha256:89b4a75bc2d8500f15463747507c9623df43886c134463e7f0527e70900e7a7b 2024-03-22T05:14:11.010 INFO:teuthology.orchestra.run.smithi005.stderr:/bin/podman: Copying blob sha256:c32ab78b488d0c72f64eded765c0cf6b5bf2c75dab66cb62a9d367fa6ec42513 2024-03-22T05:14:11.010 INFO:teuthology.orchestra.run.smithi005.stderr:/bin/podman: Copying blob sha256:a70843738bb77e1ab9c1f85969ebdfa55f178e746be081d1cb4f94011f69eb7c 2024-03-22T05:14:11.010 INFO:teuthology.orchestra.run.smithi005.stderr:/bin/podman: Copying blob sha256:599d07cb321ff0a3c82224e1138fc685793fa69b93ed5780415751a5f7e4b8c2 2024-03-22T05:14:11.010 INFO:teuthology.orchestra.run.smithi005.stderr:/bin/podman: Copying config sha256:e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-03-22T05:14:11.150 INFO:teuthology.orchestra.run.smithi005.stderr:/bin/podman: Writing manifest to image destination 2024-03-22T05:14:11.261 INFO:teuthology.orchestra.run.smithi005.stderr:/bin/podman: e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-03-22T05:14:11.724 INFO:teuthology.orchestra.run.smithi005.stderr:ceph: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-03-22T05:14:12.277 INFO:teuthology.orchestra.run.smithi005.stderr:Ceph version: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-03-22T05:14:12.277 INFO:teuthology.orchestra.run.smithi005.stderr:Extracting ceph user uid/gid from container image... 2024-03-22T05:14:12.610 INFO:teuthology.orchestra.run.smithi005.stderr:stat: 167 167 2024-03-22T05:14:13.116 INFO:teuthology.orchestra.run.smithi005.stderr:Creating initial keys... 2024-03-22T05:14:13.524 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-authtool: AQClE/1lZ5M7GhAAhNwdBFRwUzIuCfXsnDpdYg== 2024-03-22T05:14:14.413 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-authtool: AQCmE/1lmYQ9FhAAEZImjqTkSOxJ6FVYgBPc7A== 2024-03-22T05:14:15.352 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph-authtool: AQCnE/1lpWCnDxAADSu3Hq/KSd6Glpr3nVo0GQ== 2024-03-22T05:14:15.884 INFO:teuthology.orchestra.run.smithi005.stderr:Creating initial monmap... 2024-03-22T05:14:16.293 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-22T05:14:16.293 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/monmaptool: setting min_mon_release = octopus 2024-03-22T05:14:16.293 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to f59f36e2-e80a-11ee-95cd-87774f69a715 2024-03-22T05:14:16.293 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-22T05:14:16.740 INFO:teuthology.orchestra.run.smithi005.stderr:monmaptool for a [v2:172.21.15.5:3300,v1:172.21.15.5:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-22T05:14:16.740 INFO:teuthology.orchestra.run.smithi005.stderr:setting min_mon_release = octopus 2024-03-22T05:14:16.740 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/monmaptool: set fsid to f59f36e2-e80a-11ee-95cd-87774f69a715 2024-03-22T05:14:16.740 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-22T05:14:16.740 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-03-22T05:14:16.741 INFO:teuthology.orchestra.run.smithi005.stderr:Creating mon... 2024-03-22T05:14:17.655 INFO:teuthology.orchestra.run.smithi005.stderr:create mon.a on 2024-03-22T05:14:17.854 INFO:teuthology.orchestra.run.smithi005.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-22T05:14:17.998 INFO:teuthology.orchestra.run.smithi005.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-f59f36e2-e80a-11ee-95cd-87774f69a715.target → /etc/systemd/system/ceph-f59f36e2-e80a-11ee-95cd-87774f69a715.target. 2024-03-22T05:14:17.998 INFO:teuthology.orchestra.run.smithi005.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-f59f36e2-e80a-11ee-95cd-87774f69a715.target → /etc/systemd/system/ceph-f59f36e2-e80a-11ee-95cd-87774f69a715.target. 2024-03-22T05:14:18.280 INFO:teuthology.orchestra.run.smithi005.stderr:systemctl: Failed to reset failed state of unit ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@mon.a.service: Unit ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@mon.a.service not loaded. 2024-03-22T05:14:18.287 INFO:teuthology.orchestra.run.smithi005.stderr:systemctl: Created symlink /etc/systemd/system/ceph-f59f36e2-e80a-11ee-95cd-87774f69a715.target.wants/ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@mon.a.service → /etc/systemd/system/ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@.service. 2024-03-22T05:14:18.936 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:18 smithi005 podman[24429]: 2024-03-22 05:14:18.875866121 +0000 UTC m=+0.315079197 container start 22fce1753e992c05aea5e7b69953de412ad08f16cda7a4aa6c5647e3d98d7d2b (image=quay.io/ceph/ceph:v17.2.0, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mon-a, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_CLEAN=True, vcs-type=git, com.redhat.component=centos-stream-container, name=centos-stream, ceph=True, build-date=2022-05-03T08:36:31.336870, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, version=8, architecture=x86_64, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux ) 2024-03-22T05:14:18.936 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:18 smithi005 ceph-mon[24447]: mkfs f59f36e2-e80a-11ee-95cd-87774f69a715 2024-03-22T05:14:18.976 INFO:teuthology.orchestra.run.smithi005.stderr:systemctl: disabled 2024-03-22T05:14:18.983 INFO:teuthology.orchestra.run.smithi005.stderr:systemctl: inactive 2024-03-22T05:14:18.983 INFO:teuthology.orchestra.run.smithi005.stderr:firewalld.service is not enabled 2024-03-22T05:14:18.983 INFO:teuthology.orchestra.run.smithi005.stderr:Not possible to enable service . firewalld.service is not available 2024-03-22T05:14:18.984 INFO:teuthology.orchestra.run.smithi005.stderr:Waiting for mon to start... 2024-03-22T05:14:18.984 INFO:teuthology.orchestra.run.smithi005.stderr:Waiting for mon... 2024-03-22T05:14:19.195 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:18 smithi005 ceph-mon[24447]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-22T05:14:19.195 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:18 smithi005 bash[24429]: 22fce1753e992c05aea5e7b69953de412ad08f16cda7a4aa6c5647e3d98d7d2b 2024-03-22T05:14:19.196 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:18 smithi005 systemd[1]: Started Ceph mon.a for f59f36e2-e80a-11ee-95cd-87774f69a715. 2024-03-22T05:14:19.471 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: cluster: 2024-03-22T05:14:19.472 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: id: f59f36e2-e80a-11ee-95cd-87774f69a715 2024-03-22T05:14:19.472 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-03-22T05:14:19.472 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2024-03-22T05:14:19.472 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: services: 2024-03-22T05:14:19.472 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: mon: 1 daemons, quorum a (age 0.536041s) 2024-03-22T05:14:19.472 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: mgr: no daemons active 2024-03-22T05:14:19.472 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-03-22T05:14:19.472 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2024-03-22T05:14:19.472 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: data: 2024-03-22T05:14:19.472 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-03-22T05:14:19.472 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-03-22T05:14:19.472 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-03-22T05:14:19.472 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: pgs: 2024-03-22T05:14:19.472 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2024-03-22T05:14:19.931 INFO:teuthology.orchestra.run.smithi005.stderr:mon is available 2024-03-22T05:14:19.931 INFO:teuthology.orchestra.run.smithi005.stderr:Assimilating anything we can from ceph.conf... 2024-03-22T05:14:20.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:19 smithi005 ceph-mon[24447]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-22T05:14:20.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:19 smithi005 ceph-mon[24447]: monmap e1: 1 mons at {a=[v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0]} 2024-03-22T05:14:20.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:19 smithi005 ceph-mon[24447]: fsmap 2024-03-22T05:14:20.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:19 smithi005 ceph-mon[24447]: osdmap e1: 0 total, 0 up, 0 in 2024-03-22T05:14:20.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:19 smithi005 ceph-mon[24447]: mgrmap e1: no daemons active 2024-03-22T05:14:20.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:19 smithi005 ceph-mon[24447]: from='client.? 172.21.15.5:0/3504262694' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-22T05:14:20.434 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2024-03-22T05:14:20.434 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: [global] 2024-03-22T05:14:20.434 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: fsid = f59f36e2-e80a-11ee-95cd-87774f69a715 2024-03-22T05:14:20.434 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.5:3300,v1:172.21.15.5:6789] 2024-03-22T05:14:20.434 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-03-22T05:14:20.434 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-03-22T05:14:20.434 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-03-22T05:14:20.434 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-03-22T05:14:20.435 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2024-03-22T05:14:20.435 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: [mgr] 2024-03-22T05:14:20.435 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: mgr/cephadm/use_agent = False 2024-03-22T05:14:20.435 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-03-22T05:14:20.435 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2024-03-22T05:14:20.435 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: [osd] 2024-03-22T05:14:20.435 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-03-22T05:14:20.435 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-03-22T05:14:20.929 INFO:teuthology.orchestra.run.smithi005.stderr:Generating new minimal ceph.conf... 2024-03-22T05:14:21.019 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:20 smithi005 ceph-mon[24447]: from='client.? 172.21.15.5:0/2267332910' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-22T05:14:21.019 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:20 smithi005 ceph-mon[24447]: from='client.? 172.21.15.5:0/2267332910' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-22T05:14:21.943 INFO:teuthology.orchestra.run.smithi005.stderr:Restarting the monitor... 2024-03-22T05:14:21.990 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:21 smithi005 ceph-mon[24447]: from='client.? 172.21.15.5:0/996602868' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:14:21.990 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:21 smithi005 systemd[1]: Stopping Ceph mon.a for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:14:22.385 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:21 smithi005 bash[24667]: Error: no container with name or ID "ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mon.a" found: no such container 2024-03-22T05:14:22.385 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:22 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mon-a[24443]: 2024-03-22T05:14:22.085+0000 7ff8a69ee700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T05:14:22.385 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:22 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mon-a[24443]: 2024-03-22T05:14:22.085+0000 7ff8a69ee700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-22T05:14:22.385 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:22 smithi005 podman[24679]: 2024-03-22 05:14:22.1581254 +0000 UTC m=+0.158981625 container died 22fce1753e992c05aea5e7b69953de412ad08f16cda7a4aa6c5647e3d98d7d2b (image=quay.io/ceph/ceph:v17.2.0, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mon-a, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., version=8, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, RELEASE=HEAD, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, name=centos-stream, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD) 2024-03-22T05:14:22.385 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:22 smithi005 podman[24679]: 2024-03-22 05:14:22.385790155 +0000 UTC m=+0.386646371 container cleanup 22fce1753e992c05aea5e7b69953de412ad08f16cda7a4aa6c5647e3d98d7d2b (image=quay.io/ceph/ceph:v17.2.0, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mon-a, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, io.buildah.version=1.19.8, ceph=True, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, maintainer=Guillaume Abrioux , distribution-scope=public, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-03-22T05:14:22.386 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:22 smithi005 bash[24679]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mon-a 2024-03-22T05:14:22.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:22 smithi005 bash[24706]: Error: no container with name or ID "ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mon.a" found: no such container 2024-03-22T05:14:22.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:22 smithi005 podman[24691]: 2024-03-22 05:14:22.553353658 +0000 UTC m=+0.392059091 container remove 22fce1753e992c05aea5e7b69953de412ad08f16cda7a4aa6c5647e3d98d7d2b (image=quay.io/ceph/ceph:v17.2.0, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mon-a, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, release=754, build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, vcs-type=git, io.buildah.version=1.19.8, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, RELEASE=HEAD, io.openshift.expose-services=, distribution-scope=public, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, name=centos-stream, architecture=x86_64) 2024-03-22T05:14:22.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:22 smithi005 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-22T05:14:22.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:22 smithi005 systemd[1]: Stopped Ceph mon.a for f59f36e2-e80a-11ee-95cd-87774f69a715. 2024-03-22T05:14:22.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:22 smithi005 systemd[1]: Starting Ceph mon.a for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:14:23.042 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:22 smithi005 podman[24775]: 2024-03-22 05:14:22.692720806 +0000 UTC m=+0.017956991 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph:v17.2.0 2024-03-22T05:14:23.042 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:22 smithi005 podman[24775]: 2024-03-22 05:14:22.820978861 +0000 UTC m=+0.146215025 container create 38e0bba4bb0324623e0b8652e83145cb7b3fa3105e16aff7dcc8c740ed0838d3 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mon-a, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, name=centos-stream, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870) 2024-03-22T05:14:23.043 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 podman[24775]: 2024-03-22 05:14:23.016384593 +0000 UTC m=+0.341620769 container init 38e0bba4bb0324623e0b8652e83145cb7b3fa3105e16aff7dcc8c740ed0838d3 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mon-a, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, ceph=True, architecture=x86_64, GIT_CLEAN=True, name=centos-stream, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-type=git, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, release=754, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-03-22T05:14:23.043 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 podman[24775]: 2024-03-22 05:14:23.01952154 +0000 UTC m=+0.344757707 container start 38e0bba4bb0324623e0b8652e83145cb7b3fa3105e16aff7dcc8c740ed0838d3 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mon-a, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, release=754, GIT_BRANCH=HEAD, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, vcs-type=git, io.openshift.expose-services=, version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_CLEAN=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , name=centos-stream, build-date=2022-05-03T08:36:31.336870, architecture=x86_64) 2024-03-22T05:14:23.043 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: set uid:gid to 167:167 (ceph:ceph) 2024-03-22T05:14:23.043 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-03-22T05:14:23.043 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: pidfile_write: ignore empty --pid-file 2024-03-22T05:14:23.043 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: load: jerasure load: lrc 2024-03-22T05:14:23.084 INFO:teuthology.orchestra.run.smithi005.stderr:Setting mon public_network to 172.21.0.0/20 2024-03-22T05:14:23.293 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: RocksDB version: 6.15.5 2024-03-22T05:14:23.293 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-03-22T05:14:23.294 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Compile date Apr 18 2022 2024-03-22T05:14:23.294 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: DB SUMMARY 2024-03-22T05:14:23.294 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: DB Session ID: XX4CX563FF9BP2SHV45C 2024-03-22T05:14:23.294 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: CURRENT file: CURRENT 2024-03-22T05:14:23.294 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: IDENTITY file: IDENTITY 2024-03-22T05:14:23.294 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-03-22T05:14:23.294 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-22T05:14:23.294 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000010.log size: 82605 ; 2024-03-22T05:14:23.294 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.error_if_exists: 0 2024-03-22T05:14:23.294 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.create_if_missing: 0 2024-03-22T05:14:23.294 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.paranoid_checks: 1 2024-03-22T05:14:23.295 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-22T05:14:23.295 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.env: 0x5583248c6860 2024-03-22T05:14:23.295 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.fs: Posix File System 2024-03-22T05:14:23.295 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.info_log: 0x55832634ddc0 2024-03-22T05:14:23.295 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.max_file_opening_threads: 16 2024-03-22T05:14:23.295 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.statistics: (nil) 2024-03-22T05:14:23.295 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.use_fsync: 0 2024-03-22T05:14:23.295 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.max_log_file_size: 0 2024-03-22T05:14:23.295 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-22T05:14:23.295 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-22T05:14:23.295 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.keep_log_file_num: 1000 2024-03-22T05:14:23.295 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.recycle_log_file_num: 0 2024-03-22T05:14:23.295 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.allow_fallocate: 1 2024-03-22T05:14:23.295 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.allow_mmap_reads: 0 2024-03-22T05:14:23.296 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.allow_mmap_writes: 0 2024-03-22T05:14:23.296 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.use_direct_reads: 0 2024-03-22T05:14:23.296 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-22T05:14:23.296 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.create_missing_column_families: 0 2024-03-22T05:14:23.296 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.db_log_dir: 2024-03-22T05:14:23.296 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2024-03-22T05:14:23.296 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-22T05:14:23.296 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-22T05:14:23.296 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-22T05:14:23.296 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-22T05:14:23.296 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-22T05:14:23.296 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-22T05:14:23.296 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.advise_random_on_open: 1 2024-03-22T05:14:23.296 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.db_write_buffer_size: 0 2024-03-22T05:14:23.297 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.write_buffer_manager: 0x55832643e2a0 2024-03-22T05:14:23.297 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-22T05:14:23.297 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-03-22T05:14:23.297 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-22T05:14:23.297 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-22T05:14:23.297 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.rate_limiter: (nil) 2024-03-22T05:14:23.297 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-22T05:14:23.297 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.wal_recovery_mode: 2 2024-03-22T05:14:23.297 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.enable_thread_tracking: 0 2024-03-22T05:14:23.297 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.enable_pipelined_write: 0 2024-03-22T05:14:23.297 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.unordered_write: 0 2024-03-22T05:14:23.297 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-22T05:14:23.298 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-22T05:14:23.298 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-22T05:14:23.298 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-22T05:14:23.298 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.row_cache: None 2024-03-22T05:14:23.298 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.wal_filter: None 2024-03-22T05:14:23.298 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-22T05:14:23.298 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.allow_ingest_behind: 0 2024-03-22T05:14:23.298 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.preserve_deletes: 0 2024-03-22T05:14:23.298 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.two_write_queues: 0 2024-03-22T05:14:23.299 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.manual_wal_flush: 0 2024-03-22T05:14:23.299 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.atomic_flush: 0 2024-03-22T05:14:23.299 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-22T05:14:23.299 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-22T05:14:23.299 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-22T05:14:23.299 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.log_readahead_size: 0 2024-03-22T05:14:23.299 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-22T05:14:23.299 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.best_efforts_recovery: 0 2024-03-22T05:14:23.299 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-22T05:14:23.299 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-22T05:14:23.300 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.allow_data_in_errors: 0 2024-03-22T05:14:23.300 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.db_host_id: __hostname__ 2024-03-22T05:14:23.300 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.max_background_jobs: 2 2024-03-22T05:14:23.300 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.max_background_compactions: -1 2024-03-22T05:14:23.300 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.max_subcompactions: 1 2024-03-22T05:14:23.300 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-22T05:14:23.300 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-22T05:14:23.301 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-22T05:14:23.302 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.max_total_wal_size: 0 2024-03-22T05:14:23.302 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-22T05:14:23.302 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-22T05:14:23.302 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-22T05:14:23.302 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-22T05:14:23.302 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.max_open_files: -1 2024-03-22T05:14:23.302 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.bytes_per_sync: 0 2024-03-22T05:14:23.302 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-22T05:14:23.302 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-22T05:14:23.302 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.compaction_readahead_size: 0 2024-03-22T05:14:23.302 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.max_background_flushes: -1 2024-03-22T05:14:23.302 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Compression algorithms supported: 2024-03-22T05:14:23.302 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-22T05:14:23.303 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: kZSTD supported: 0 2024-03-22T05:14:23.303 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: kXpressCompression supported: 0 2024-03-22T05:14:23.303 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: kLZ4HCCompression supported: 1 2024-03-22T05:14:23.303 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: kLZ4Compression supported: 1 2024-03-22T05:14:23.303 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: kBZip2Compression supported: 0 2024-03-22T05:14:23.303 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: kZlibCompression supported: 1 2024-03-22T05:14:23.303 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: kSnappyCompression supported: 1 2024-03-22T05:14:23.303 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-22T05:14:23.303 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 2024-03-22T05:14:23.303 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-03-22T05:14:23.303 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-22T05:14:23.303 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.merge_operator: 2024-03-22T05:14:23.303 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.compaction_filter: None 2024-03-22T05:14:23.303 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.compaction_filter_factory: None 2024-03-22T05:14:23.303 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.sst_partitioner_factory: None 2024-03-22T05:14:23.304 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-22T05:14:23.304 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-22T05:14:23.304 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x558326319d00) 2024-03-22T05:14:23.304 INFO:journalctl@ceph.mon.a.smithi005.stdout: cache_index_and_filter_blocks: 1 2024-03-22T05:14:23.304 INFO:journalctl@ceph.mon.a.smithi005.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-22T05:14:23.304 INFO:journalctl@ceph.mon.a.smithi005.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-22T05:14:23.304 INFO:journalctl@ceph.mon.a.smithi005.stdout: pin_top_level_index_and_filter: 1 2024-03-22T05:14:23.304 INFO:journalctl@ceph.mon.a.smithi005.stdout: index_type: 0 2024-03-22T05:14:23.304 INFO:journalctl@ceph.mon.a.smithi005.stdout: data_block_index_type: 0 2024-03-22T05:14:23.304 INFO:journalctl@ceph.mon.a.smithi005.stdout: index_shortening: 1 2024-03-22T05:14:23.304 INFO:journalctl@ceph.mon.a.smithi005.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-22T05:14:23.304 INFO:journalctl@ceph.mon.a.smithi005.stdout: hash_index_allow_collision: 1 2024-03-22T05:14:23.304 INFO:journalctl@ceph.mon.a.smithi005.stdout: checksum: 1 2024-03-22T05:14:23.305 INFO:journalctl@ceph.mon.a.smithi005.stdout: no_block_cache: 0 2024-03-22T05:14:23.305 INFO:journalctl@ceph.mon.a.smithi005.stdout: block_cache: 0x558326384170 2024-03-22T05:14:23.305 INFO:journalctl@ceph.mon.a.smithi005.stdout: block_cache_name: BinnedLRUCache 2024-03-22T05:14:23.305 INFO:journalctl@ceph.mon.a.smithi005.stdout: block_cache_options: 2024-03-22T05:14:23.305 INFO:journalctl@ceph.mon.a.smithi005.stdout: capacity : 536870912 2024-03-22T05:14:23.305 INFO:journalctl@ceph.mon.a.smithi005.stdout: num_shard_bits : 4 2024-03-22T05:14:23.305 INFO:journalctl@ceph.mon.a.smithi005.stdout: strict_capacity_limit : 0 2024-03-22T05:14:23.305 INFO:journalctl@ceph.mon.a.smithi005.stdout: high_pri_pool_ratio: 0.000 2024-03-22T05:14:23.305 INFO:journalctl@ceph.mon.a.smithi005.stdout: block_cache_compressed: (nil) 2024-03-22T05:14:23.305 INFO:journalctl@ceph.mon.a.smithi005.stdout: persistent_cache: (nil) 2024-03-22T05:14:23.305 INFO:journalctl@ceph.mon.a.smithi005.stdout: block_size: 4096 2024-03-22T05:14:23.305 INFO:journalctl@ceph.mon.a.smithi005.stdout: block_size_deviation: 10 2024-03-22T05:14:23.305 INFO:journalctl@ceph.mon.a.smithi005.stdout: block_restart_interval: 16 2024-03-22T05:14:23.305 INFO:journalctl@ceph.mon.a.smithi005.stdout: index_block_restart_interval: 1 2024-03-22T05:14:23.306 INFO:journalctl@ceph.mon.a.smithi005.stdout: metadata_block_size: 4096 2024-03-22T05:14:23.306 INFO:journalctl@ceph.mon.a.smithi005.stdout: partition_filters: 0 2024-03-22T05:14:23.306 INFO:journalctl@ceph.mon.a.smithi005.stdout: use_delta_encoding: 1 2024-03-22T05:14:23.306 INFO:journalctl@ceph.mon.a.smithi005.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-03-22T05:14:23.306 INFO:journalctl@ceph.mon.a.smithi005.stdout: whole_key_filtering: 1 2024-03-22T05:14:23.306 INFO:journalctl@ceph.mon.a.smithi005.stdout: verify_compression: 0 2024-03-22T05:14:23.306 INFO:journalctl@ceph.mon.a.smithi005.stdout: read_amp_bytes_per_bit: 0 2024-03-22T05:14:23.306 INFO:journalctl@ceph.mon.a.smithi005.stdout: format_version: 4 2024-03-22T05:14:23.306 INFO:journalctl@ceph.mon.a.smithi005.stdout: enable_index_compression: 1 2024-03-22T05:14:23.306 INFO:journalctl@ceph.mon.a.smithi005.stdout: block_align: 0 2024-03-22T05:14:23.306 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.write_buffer_size: 33554432 2024-03-22T05:14:23.306 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.max_write_buffer_number: 2 2024-03-22T05:14:23.306 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.compression: NoCompression 2024-03-22T05:14:23.307 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.bottommost_compression: Disabled 2024-03-22T05:14:23.307 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.prefix_extractor: nullptr 2024-03-22T05:14:23.307 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-22T05:14:23.307 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.num_levels: 7 2024-03-22T05:14:23.307 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-22T05:14:23.307 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-22T05:14:23.307 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-22T05:14:23.307 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-22T05:14:23.307 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-22T05:14:23.307 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-22T05:14:23.307 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-22T05:14:23.307 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-22T05:14:23.307 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-22T05:14:23.307 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-22T05:14:23.308 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-22T05:14:23.308 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.compression_opts.level: 32767 2024-03-22T05:14:23.308 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.compression_opts.strategy: 0 2024-03-22T05:14:23.308 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-22T05:14:23.308 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-22T05:14:23.308 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-22T05:14:23.308 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.compression_opts.enabled: false 2024-03-22T05:14:23.308 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-22T05:14:23.308 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-22T05:14:23.308 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-22T05:14:23.308 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.target_file_size_base: 67108864 2024-03-22T05:14:23.308 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-22T05:14:23.308 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-22T05:14:23.308 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-22T05:14:23.309 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-22T05:14:23.309 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-22T05:14:23.309 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-22T05:14:23.309 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-22T05:14:23.309 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-22T05:14:23.309 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-22T05:14:23.309 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-22T05:14:23.309 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-22T05:14:23.309 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-22T05:14:23.309 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-22T05:14:23.309 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.arena_block_size: 4194304 2024-03-22T05:14:23.309 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-22T05:14:23.309 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-22T05:14:23.309 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-03-22T05:14:23.309 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.disable_auto_compactions: 0 2024-03-22T05:14:23.310 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-22T05:14:23.310 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-22T05:14:23.310 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-22T05:14:23.310 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-22T05:14:23.310 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-22T05:14:23.310 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-22T05:14:23.310 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-22T05:14:23.310 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-22T05:14:23.310 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-22T05:14:23.311 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-22T05:14:23.311 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.table_properties_collectors: 2024-03-22T05:14:23.311 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.inplace_update_support: 0 2024-03-22T05:14:23.311 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-22T05:14:23.311 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-22T05:14:23.311 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-22T05:14:23.311 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-22T05:14:23.311 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.bloom_locality: 0 2024-03-22T05:14:23.311 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.max_successive_merges: 0 2024-03-22T05:14:23.311 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-22T05:14:23.311 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.paranoid_file_checks: 0 2024-03-22T05:14:23.311 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.force_consistency_checks: 1 2024-03-22T05:14:23.311 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.report_bg_io_stats: 0 2024-03-22T05:14:23.311 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.ttl: 2592000 2024-03-22T05:14:23.312 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-22T05:14:23.312 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.enable_blob_files: false 2024-03-22T05:14:23.312 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.min_blob_size: 0 2024-03-22T05:14:23.312 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.blob_file_size: 268435456 2024-03-22T05:14:23.312 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-22T05:14:23.312 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-22T05:14:23.312 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-22T05:14:23.312 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-22T05:14:23.312 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 5 2024-03-22T05:14:23.312 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: [db/version_set.cc:4083] Creating manifest 13 2024-03-22T05:14:23.312 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711084463044959, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-03-22T05:14:23.312 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #10 mode 2 2024-03-22T05:14:23.312 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-03-22T05:14:23.312 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711084463045694, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 79155, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 77418, "index_size": 225, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10027, "raw_average_key_size": 47, "raw_value_size": 71727, "raw_average_value_size": 339, "num_data_blocks": 10, "num_entries": 211, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1711084463, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "5e89885b-5fdf-4fe0-8169-5b6a3e97d4f9", "db_session_id": "XX4CX563FF9BP2SHV45C"}} 2024-03-22T05:14:23.313 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: [db/version_set.cc:4083] Creating manifest 15 2024-03-22T05:14:23.313 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711084463046024, "job": 1, "event": "recovery_finished"} 2024-03-22T05:14:23.313 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-22T05:14:23.313 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x55832636aa80 2024-03-22T05:14:23.313 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: DB pointer 0x55832637a000 2024-03-22T05:14:23.313 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-03-22T05:14:23.313 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-03-22T05:14:23.313 INFO:journalctl@ceph.mon.a.smithi005.stdout: ** DB Stats ** 2024-03-22T05:14:23.313 INFO:journalctl@ceph.mon.a.smithi005.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-22T05:14:23.313 INFO:journalctl@ceph.mon.a.smithi005.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-22T05:14:23.313 INFO:journalctl@ceph.mon.a.smithi005.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-22T05:14:23.313 INFO:journalctl@ceph.mon.a.smithi005.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-22T05:14:23.313 INFO:journalctl@ceph.mon.a.smithi005.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-22T05:14:23.313 INFO:journalctl@ceph.mon.a.smithi005.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-03-22T05:14:23.313 INFO:journalctl@ceph.mon.a.smithi005.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-22T05:14:23.314 INFO:journalctl@ceph.mon.a.smithi005.stdout: 2024-03-22T05:14:23.314 INFO:journalctl@ceph.mon.a.smithi005.stdout: ** Compaction Stats [default] ** 2024-03-22T05:14:23.314 INFO:journalctl@ceph.mon.a.smithi005.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-22T05:14:23.314 INFO:journalctl@ceph.mon.a.smithi005.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-22T05:14:23.314 INFO:journalctl@ceph.mon.a.smithi005.stdout: L0 2/0 79.06 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 148.3 0.00 0.00 1 0.001 0 0 2024-03-22T05:14:23.314 INFO:journalctl@ceph.mon.a.smithi005.stdout: Sum 2/0 79.06 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 148.3 0.00 0.00 1 0.001 0 0 2024-03-22T05:14:23.314 INFO:journalctl@ceph.mon.a.smithi005.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 148.3 0.00 0.00 1 0.001 0 0 2024-03-22T05:14:23.314 INFO:journalctl@ceph.mon.a.smithi005.stdout: 2024-03-22T05:14:23.314 INFO:journalctl@ceph.mon.a.smithi005.stdout: ** Compaction Stats [default] ** 2024-03-22T05:14:23.314 INFO:journalctl@ceph.mon.a.smithi005.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-22T05:14:23.314 INFO:journalctl@ceph.mon.a.smithi005.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-22T05:14:23.314 INFO:journalctl@ceph.mon.a.smithi005.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 148.3 0.00 0.00 1 0.001 0 0 2024-03-22T05:14:23.314 INFO:journalctl@ceph.mon.a.smithi005.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-22T05:14:23.314 INFO:journalctl@ceph.mon.a.smithi005.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-22T05:14:23.315 INFO:journalctl@ceph.mon.a.smithi005.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-22T05:14:23.315 INFO:journalctl@ceph.mon.a.smithi005.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-22T05:14:23.315 INFO:journalctl@ceph.mon.a.smithi005.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-22T05:14:23.315 INFO:journalctl@ceph.mon.a.smithi005.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-22T05:14:23.315 INFO:journalctl@ceph.mon.a.smithi005.stdout: Cumulative compaction: 0.00 GB write, 25.85 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-22T05:14:23.315 INFO:journalctl@ceph.mon.a.smithi005.stdout: Interval compaction: 0.00 GB write, 25.85 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-22T05:14:23.315 INFO:journalctl@ceph.mon.a.smithi005.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-22T05:14:23.315 INFO:journalctl@ceph.mon.a.smithi005.stdout: 2024-03-22T05:14:23.315 INFO:journalctl@ceph.mon.a.smithi005.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-22T05:14:23.315 INFO:journalctl@ceph.mon.a.smithi005.stdout: 2024-03-22T05:14:23.315 INFO:journalctl@ceph.mon.a.smithi005.stdout: ** Compaction Stats [default] ** 2024-03-22T05:14:23.315 INFO:journalctl@ceph.mon.a.smithi005.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-22T05:14:23.315 INFO:journalctl@ceph.mon.a.smithi005.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-22T05:14:23.315 INFO:journalctl@ceph.mon.a.smithi005.stdout: L0 2/0 79.06 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 148.3 0.00 0.00 1 0.001 0 0 2024-03-22T05:14:23.316 INFO:journalctl@ceph.mon.a.smithi005.stdout: Sum 2/0 79.06 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 148.3 0.00 0.00 1 0.001 0 0 2024-03-22T05:14:23.316 INFO:journalctl@ceph.mon.a.smithi005.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-03-22T05:14:23.316 INFO:journalctl@ceph.mon.a.smithi005.stdout: 2024-03-22T05:14:23.316 INFO:journalctl@ceph.mon.a.smithi005.stdout: ** Compaction Stats [default] ** 2024-03-22T05:14:23.316 INFO:journalctl@ceph.mon.a.smithi005.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-22T05:14:23.316 INFO:journalctl@ceph.mon.a.smithi005.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-22T05:14:23.316 INFO:journalctl@ceph.mon.a.smithi005.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 148.3 0.00 0.00 1 0.001 0 0 2024-03-22T05:14:23.316 INFO:journalctl@ceph.mon.a.smithi005.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-22T05:14:23.316 INFO:journalctl@ceph.mon.a.smithi005.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-22T05:14:23.316 INFO:journalctl@ceph.mon.a.smithi005.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-22T05:14:23.316 INFO:journalctl@ceph.mon.a.smithi005.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-22T05:14:23.316 INFO:journalctl@ceph.mon.a.smithi005.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-22T05:14:23.316 INFO:journalctl@ceph.mon.a.smithi005.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-22T05:14:23.316 INFO:journalctl@ceph.mon.a.smithi005.stdout: Cumulative compaction: 0.00 GB write, 25.58 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-22T05:14:23.316 INFO:journalctl@ceph.mon.a.smithi005.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-22T05:14:23.317 INFO:journalctl@ceph.mon.a.smithi005.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-22T05:14:23.317 INFO:journalctl@ceph.mon.a.smithi005.stdout: 2024-03-22T05:14:23.317 INFO:journalctl@ceph.mon.a.smithi005.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-22T05:14:23.317 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: starting mon.a rank 0 at public addrs [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] at bind addrs [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid f59f36e2-e80a-11ee-95cd-87774f69a715 2024-03-22T05:14:23.317 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: mon.a@-1(???) e1 preinit fsid f59f36e2-e80a-11ee-95cd-87774f69a715 2024-03-22T05:14:23.317 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: mon.a@-1(???).mds e1 new map 2024-03-22T05:14:23.317 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: mon.a@-1(???).mds e1 print_map 2024-03-22T05:14:23.317 INFO:journalctl@ceph.mon.a.smithi005.stdout: e1 2024-03-22T05:14:23.317 INFO:journalctl@ceph.mon.a.smithi005.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-22T05:14:23.317 INFO:journalctl@ceph.mon.a.smithi005.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T05:14:23.317 INFO:journalctl@ceph.mon.a.smithi005.stdout: legacy client fscid: -1 2024-03-22T05:14:23.317 INFO:journalctl@ceph.mon.a.smithi005.stdout: 2024-03-22T05:14:23.317 INFO:journalctl@ceph.mon.a.smithi005.stdout: No filesystems configured 2024-03-22T05:14:23.317 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-22T05:14:23.317 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-22T05:14:23.318 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-22T05:14:23.318 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-22T05:14:23.318 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-22T05:14:23.318 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 bash[24775]: 38e0bba4bb0324623e0b8652e83145cb7b3fa3105e16aff7dcc8c740ed0838d3 2024-03-22T05:14:23.318 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 systemd[1]: Started Ceph mon.a for f59f36e2-e80a-11ee-95cd-87774f69a715. 2024-03-22T05:14:23.318 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-22T05:14:23.318 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: monmap e1: 1 mons at {a=[v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0]} 2024-03-22T05:14:23.318 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: fsmap 2024-03-22T05:14:23.318 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: osdmap e1: 0 total, 0 up, 0 in 2024-03-22T05:14:23.318 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:23 smithi005 ceph-mon[24793]: mgrmap e1: no daemons active 2024-03-22T05:14:24.091 INFO:teuthology.orchestra.run.smithi005.stderr:Wrote config to /etc/ceph/ceph.conf 2024-03-22T05:14:24.091 INFO:teuthology.orchestra.run.smithi005.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-22T05:14:24.091 INFO:teuthology.orchestra.run.smithi005.stderr:Creating mgr... 2024-03-22T05:14:24.091 INFO:teuthology.orchestra.run.smithi005.stderr:Verifying port 9283 ... 2024-03-22T05:14:24.260 INFO:teuthology.orchestra.run.smithi005.stderr:systemctl: Failed to reset failed state of unit ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@mgr.y.service: Unit ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@mgr.y.service not loaded. 2024-03-22T05:14:24.266 INFO:teuthology.orchestra.run.smithi005.stderr:systemctl: Created symlink /etc/systemd/system/ceph-f59f36e2-e80a-11ee-95cd-87774f69a715.target.wants/ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@mgr.y.service → /etc/systemd/system/ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@.service. 2024-03-22T05:14:24.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:24 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/1161200411' entity='client.admin' 2024-03-22T05:14:24.703 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:14:24 smithi005 podman[25006]: 2024-03-22 05:14:24.534652564 +0000 UTC m=+0.018121197 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph:v17.2.0 2024-03-22T05:14:24.914 INFO:teuthology.orchestra.run.smithi005.stderr:systemctl: disabled 2024-03-22T05:14:24.922 INFO:teuthology.orchestra.run.smithi005.stderr:systemctl: inactive 2024-03-22T05:14:24.923 INFO:teuthology.orchestra.run.smithi005.stderr:firewalld.service is not enabled 2024-03-22T05:14:24.923 INFO:teuthology.orchestra.run.smithi005.stderr:Not possible to enable service . firewalld.service is not available 2024-03-22T05:14:24.933 INFO:teuthology.orchestra.run.smithi005.stderr:systemctl: disabled 2024-03-22T05:14:24.940 INFO:teuthology.orchestra.run.smithi005.stderr:systemctl: inactive 2024-03-22T05:14:24.941 INFO:teuthology.orchestra.run.smithi005.stderr:firewalld.service is not enabled 2024-03-22T05:14:24.941 INFO:teuthology.orchestra.run.smithi005.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-03-22T05:14:24.941 INFO:teuthology.orchestra.run.smithi005.stderr:Waiting for mgr to start... 2024-03-22T05:14:24.941 INFO:teuthology.orchestra.run.smithi005.stderr:Waiting for mgr... 2024-03-22T05:14:25.059 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:14:24 smithi005 podman[25006]: 2024-03-22 05:14:24.817048139 +0000 UTC m=+0.300516769 container init fcee4f81e3eeaaaed6b6a2392a8a8419457fcee212e047709c4902351ae4786d (image=quay.io/ceph/ceph:v17.2.0, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, GIT_BRANCH=HEAD, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, distribution-scope=public, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, version=8, maintainer=Guillaume Abrioux , architecture=x86_64, GIT_CLEAN=True, name=centos-stream) 2024-03-22T05:14:25.060 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:14:24 smithi005 podman[25006]: 2024-03-22 05:14:24.819787641 +0000 UTC m=+0.303256266 container start fcee4f81e3eeaaaed6b6a2392a8a8419457fcee212e047709c4902351ae4786d (image=quay.io/ceph/ceph:v17.2.0, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y, com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, distribution-scope=public, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True, io.openshift.expose-services=, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, io.openshift.tags=base centos centos-stream, ceph=True, name=centos-stream) 2024-03-22T05:14:25.060 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:14:24 smithi005 bash[25006]: fcee4f81e3eeaaaed6b6a2392a8a8419457fcee212e047709c4902351ae4786d 2024-03-22T05:14:25.060 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:14:24 smithi005 systemd[1]: Started Ceph mgr.y for f59f36e2-e80a-11ee-95cd-87774f69a715. 2024-03-22T05:14:25.448 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2024-03-22T05:14:25.448 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: { 2024-03-22T05:14:25.448 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "fsid": "f59f36e2-e80a-11ee-95cd-87774f69a715", 2024-03-22T05:14:25.448 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "health": { 2024-03-22T05:14:25.448 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-03-22T05:14:25.448 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "checks": {}, 2024-03-22T05:14:25.448 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "mutes": [] 2024-03-22T05:14:25.448 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: }, 2024-03-22T05:14:25.448 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-03-22T05:14:25.448 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "quorum": [ 2024-03-22T05:14:25.448 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 0 2024-03-22T05:14:25.449 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: ], 2024-03-22T05:14:25.449 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "quorum_names": [ 2024-03-22T05:14:25.449 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "a" 2024-03-22T05:14:25.449 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: ], 2024-03-22T05:14:25.449 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "quorum_age": 2, 2024-03-22T05:14:25.449 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "monmap": { 2024-03-22T05:14:25.449 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-22T05:14:25.449 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-03-22T05:14:25.449 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "num_mons": 1 2024-03-22T05:14:25.449 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: }, 2024-03-22T05:14:25.449 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "osdmap": { 2024-03-22T05:14:25.449 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-22T05:14:25.449 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "num_osds": 0, 2024-03-22T05:14:25.449 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-03-22T05:14:25.449 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-03-22T05:14:25.450 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-03-22T05:14:25.450 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-03-22T05:14:25.450 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-03-22T05:14:25.450 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: }, 2024-03-22T05:14:25.450 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "pgmap": { 2024-03-22T05:14:25.450 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-03-22T05:14:25.450 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-03-22T05:14:25.450 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "num_pools": 0, 2024-03-22T05:14:25.450 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "num_objects": 0, 2024-03-22T05:14:25.450 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-03-22T05:14:25.450 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-03-22T05:14:25.450 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-03-22T05:14:25.450 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "bytes_total": 0 2024-03-22T05:14:25.450 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: }, 2024-03-22T05:14:25.451 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "fsmap": { 2024-03-22T05:14:25.451 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-22T05:14:25.451 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "by_rank": [], 2024-03-22T05:14:25.451 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "up:standby": 0 2024-03-22T05:14:25.451 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: }, 2024-03-22T05:14:25.451 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "mgrmap": { 2024-03-22T05:14:25.451 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "available": false, 2024-03-22T05:14:25.451 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-03-22T05:14:25.451 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "modules": [ 2024-03-22T05:14:25.451 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "iostat", 2024-03-22T05:14:25.451 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "nfs", 2024-03-22T05:14:25.451 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "restful" 2024-03-22T05:14:25.451 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: ], 2024-03-22T05:14:25.451 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "services": {} 2024-03-22T05:14:25.451 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: }, 2024-03-22T05:14:25.452 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "servicemap": { 2024-03-22T05:14:25.452 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-22T05:14:25.452 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "modified": "2024-03-22T05:14:18.934855+0000", 2024-03-22T05:14:25.452 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "services": {} 2024-03-22T05:14:25.452 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: }, 2024-03-22T05:14:25.452 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "progress_events": {} 2024-03-22T05:14:25.452 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: } 2024-03-22T05:14:25.629 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:14:25 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:14:25.334+0000 7f2dd3632000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-22T05:14:25.630 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:14:25 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:14:25.428+0000 7f2dd3632000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-22T05:14:25.947 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:25 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/1376807164' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-22T05:14:25.950 INFO:teuthology.orchestra.run.smithi005.stderr:mgr not available, waiting (1/15)... 2024-03-22T05:14:26.202 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:14:26 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:14:26.044+0000 7f2dd3632000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-22T05:14:26.202 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:14:26 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:14:26.134+0000 7f2dd3632000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-22T05:14:26.477 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:14:26 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:14:26.218+0000 7f2dd3632000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-22T05:14:26.477 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:14:26 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:14:26.366+0000 7f2dd3632000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-22T05:14:26.477 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:14:26 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:14:26.448+0000 7f2dd3632000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-22T05:14:26.817 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:14:26 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:14:26.562+0000 7f2dd3632000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-22T05:14:26.817 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:14:26 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:14:26.648+0000 7f2dd3632000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-22T05:14:26.817 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:14:26 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:14:26.816+0000 7f2dd3632000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-22T05:14:27.202 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:14:26 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:14:26.908+0000 7f2dd3632000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-22T05:14:28.315 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:14:27 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:14:27.987+0000 7f2dd3632000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-22T05:14:28.316 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:14:28 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:14:28.121+0000 7f2dd3632000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-22T05:14:28.316 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:14:28 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:14:28.315+0000 7f2dd3632000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-22T05:14:28.702 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:14:28 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:14:28.405+0000 7f2dd3632000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-22T05:14:28.702 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:14:28 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:14:28.543+0000 7f2dd3632000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-22T05:14:29.177 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:14:28 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:14:28.868+0000 7f2dd3632000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-22T05:14:29.177 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:14:29 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:14:29.092+0000 7f2dd3632000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-22T05:14:29.177 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:14:29 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:14:29.177+0000 7f2dd3632000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-22T05:14:29.702 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:14:29 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:14:29.440+0000 7f2dd3632000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-22T05:14:30.629 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:14:30 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:14:30.374+0000 7f2dd3632000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-22T05:14:30.952 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:14:30 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:14:30.676+0000 7f2dd3632000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-22T05:14:30.952 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:14:30 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:14:30.772+0000 7f2dd3632000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-22T05:14:31.452 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:14:30 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:14:30.924+0000 7f2dd3632000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-22T05:14:31.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:30 smithi005 ceph-mon[24793]: Activating manager daemon y 2024-03-22T05:14:31.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:30 smithi005 ceph-mon[24793]: mgrmap e2: y(active, starting, since 0.0025554s) 2024-03-22T05:14:31.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:30 smithi005 ceph-mon[24793]: from='mgr.14100 172.21.15.5:0/3783068813' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-22T05:14:31.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:30 smithi005 ceph-mon[24793]: from='mgr.14100 172.21.15.5:0/3783068813' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-22T05:14:31.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:30 smithi005 ceph-mon[24793]: from='mgr.14100 172.21.15.5:0/3783068813' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-22T05:14:31.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:30 smithi005 ceph-mon[24793]: from='mgr.14100 172.21.15.5:0/3783068813' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-22T05:14:31.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:30 smithi005 ceph-mon[24793]: from='mgr.14100 172.21.15.5:0/3783068813' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-22T05:14:31.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:30 smithi005 ceph-mon[24793]: Manager daemon y is now available 2024-03-22T05:14:32.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:31 smithi005 ceph-mon[24793]: from='mgr.14100 172.21.15.5:0/3783068813' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-22T05:14:32.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:32 smithi005 ceph-mon[24793]: from='mgr.14100 172.21.15.5:0/3783068813' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-22T05:14:32.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:32 smithi005 ceph-mon[24793]: from='mgr.14100 172.21.15.5:0/3783068813' entity='mgr.y' 2024-03-22T05:14:32.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:32 smithi005 ceph-mon[24793]: from='mgr.14100 172.21.15.5:0/3783068813' entity='mgr.y' 2024-03-22T05:14:32.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:32 smithi005 ceph-mon[24793]: from='mgr.14100 172.21.15.5:0/3783068813' entity='mgr.y' 2024-03-22T05:14:32.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:32 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/2279447173' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-22T05:14:32.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:32 smithi005 ceph-mon[24793]: mgrmap e3: y(active, since 1.0061s) 2024-03-22T05:14:32.557 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2024-03-22T05:14:32.563 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: { 2024-03-22T05:14:32.563 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "fsid": "f59f36e2-e80a-11ee-95cd-87774f69a715", 2024-03-22T05:14:32.563 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "health": { 2024-03-22T05:14:32.563 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-03-22T05:14:32.563 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "checks": {}, 2024-03-22T05:14:32.563 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "mutes": [] 2024-03-22T05:14:32.563 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: }, 2024-03-22T05:14:32.563 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-03-22T05:14:32.563 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "quorum": [ 2024-03-22T05:14:32.563 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 0 2024-03-22T05:14:32.563 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: ], 2024-03-22T05:14:32.563 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "quorum_names": [ 2024-03-22T05:14:32.563 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "a" 2024-03-22T05:14:32.564 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: ], 2024-03-22T05:14:32.564 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "quorum_age": 8, 2024-03-22T05:14:32.564 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "monmap": { 2024-03-22T05:14:32.564 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-22T05:14:32.564 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-03-22T05:14:32.564 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "num_mons": 1 2024-03-22T05:14:32.564 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: }, 2024-03-22T05:14:32.564 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "osdmap": { 2024-03-22T05:14:32.564 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-22T05:14:32.564 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "num_osds": 0, 2024-03-22T05:14:32.564 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-03-22T05:14:32.564 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-03-22T05:14:32.564 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-03-22T05:14:32.564 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-03-22T05:14:32.565 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-03-22T05:14:32.565 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: }, 2024-03-22T05:14:32.565 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "pgmap": { 2024-03-22T05:14:32.565 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-03-22T05:14:32.565 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-03-22T05:14:32.565 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "num_pools": 0, 2024-03-22T05:14:32.565 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "num_objects": 0, 2024-03-22T05:14:32.565 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-03-22T05:14:32.565 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-03-22T05:14:32.565 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-03-22T05:14:32.565 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "bytes_total": 0 2024-03-22T05:14:32.565 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: }, 2024-03-22T05:14:32.565 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "fsmap": { 2024-03-22T05:14:32.565 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-22T05:14:32.565 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "by_rank": [], 2024-03-22T05:14:32.566 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "up:standby": 0 2024-03-22T05:14:32.566 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: }, 2024-03-22T05:14:32.566 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "mgrmap": { 2024-03-22T05:14:32.566 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "available": false, 2024-03-22T05:14:32.566 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-03-22T05:14:32.566 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "modules": [ 2024-03-22T05:14:32.566 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "iostat", 2024-03-22T05:14:32.566 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "nfs", 2024-03-22T05:14:32.566 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "restful" 2024-03-22T05:14:32.566 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: ], 2024-03-22T05:14:32.566 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "services": {} 2024-03-22T05:14:32.566 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: }, 2024-03-22T05:14:32.566 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "servicemap": { 2024-03-22T05:14:32.566 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-22T05:14:32.566 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "modified": "2024-03-22T05:14:18.934855+0000", 2024-03-22T05:14:32.567 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "services": {} 2024-03-22T05:14:32.567 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: }, 2024-03-22T05:14:32.567 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "progress_events": {} 2024-03-22T05:14:32.567 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: } 2024-03-22T05:14:34.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:33 smithi005 ceph-mon[24793]: mgrmap e4: y(active, since 2s) 2024-03-22T05:14:36.514 INFO:teuthology.orchestra.run.smithi005.stderr:mgr not available, waiting (2/15)... 2024-03-22T05:14:39.157 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2024-03-22T05:14:39.157 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: { 2024-03-22T05:14:39.157 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "fsid": "f59f36e2-e80a-11ee-95cd-87774f69a715", 2024-03-22T05:14:39.157 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "health": { 2024-03-22T05:14:39.157 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-03-22T05:14:39.157 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "checks": {}, 2024-03-22T05:14:39.157 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "mutes": [] 2024-03-22T05:14:39.158 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: }, 2024-03-22T05:14:39.158 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-03-22T05:14:39.158 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "quorum": [ 2024-03-22T05:14:39.158 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 0 2024-03-22T05:14:39.158 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: ], 2024-03-22T05:14:39.158 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "quorum_names": [ 2024-03-22T05:14:39.158 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "a" 2024-03-22T05:14:39.158 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: ], 2024-03-22T05:14:39.158 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "quorum_age": 16, 2024-03-22T05:14:39.158 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "monmap": { 2024-03-22T05:14:39.158 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-22T05:14:39.158 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-03-22T05:14:39.159 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "num_mons": 1 2024-03-22T05:14:39.159 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: }, 2024-03-22T05:14:39.159 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "osdmap": { 2024-03-22T05:14:39.159 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-22T05:14:39.159 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "num_osds": 0, 2024-03-22T05:14:39.159 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-03-22T05:14:39.159 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-03-22T05:14:39.159 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-03-22T05:14:39.159 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-03-22T05:14:39.159 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-03-22T05:14:39.159 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: }, 2024-03-22T05:14:39.159 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "pgmap": { 2024-03-22T05:14:39.160 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-03-22T05:14:39.160 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-03-22T05:14:39.160 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "num_pools": 0, 2024-03-22T05:14:39.160 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "num_objects": 0, 2024-03-22T05:14:39.160 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-03-22T05:14:39.160 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-03-22T05:14:39.160 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-03-22T05:14:39.160 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "bytes_total": 0 2024-03-22T05:14:39.160 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: }, 2024-03-22T05:14:39.160 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "fsmap": { 2024-03-22T05:14:39.160 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-22T05:14:39.160 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "by_rank": [], 2024-03-22T05:14:39.160 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "up:standby": 0 2024-03-22T05:14:39.161 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: }, 2024-03-22T05:14:39.161 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "mgrmap": { 2024-03-22T05:14:39.161 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "available": true, 2024-03-22T05:14:39.161 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-03-22T05:14:39.161 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "modules": [ 2024-03-22T05:14:39.161 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "iostat", 2024-03-22T05:14:39.161 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "nfs", 2024-03-22T05:14:39.161 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "restful" 2024-03-22T05:14:39.161 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: ], 2024-03-22T05:14:39.162 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "services": {} 2024-03-22T05:14:39.162 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: }, 2024-03-22T05:14:39.162 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "servicemap": { 2024-03-22T05:14:39.162 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-22T05:14:39.162 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "modified": "2024-03-22T05:14:18.934855+0000", 2024-03-22T05:14:39.162 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "services": {} 2024-03-22T05:14:39.162 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: }, 2024-03-22T05:14:39.162 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "progress_events": {} 2024-03-22T05:14:39.162 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: } 2024-03-22T05:14:39.449 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:39 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/406127823' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-22T05:14:39.728 INFO:teuthology.orchestra.run.smithi005.stderr:mgr is available 2024-03-22T05:14:40.334 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2024-03-22T05:14:40.334 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: [global] 2024-03-22T05:14:40.334 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: fsid = f59f36e2-e80a-11ee-95cd-87774f69a715 2024-03-22T05:14:40.334 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-03-22T05:14:40.334 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-03-22T05:14:40.334 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-03-22T05:14:40.334 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-03-22T05:14:40.334 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2024-03-22T05:14:40.334 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: [mgr] 2024-03-22T05:14:40.334 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-03-22T05:14:40.334 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 2024-03-22T05:14:40.334 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: [osd] 2024-03-22T05:14:40.335 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-03-22T05:14:40.335 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-03-22T05:14:40.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:40 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/2982494327' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-22T05:14:40.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:40 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/2982494327' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-22T05:14:40.850 INFO:teuthology.orchestra.run.smithi005.stderr:Enabling cephadm module... 2024-03-22T05:14:41.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:41 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/2506956309' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-22T05:14:41.952 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:14:41 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: ignoring --setuser ceph since I am not root 2024-03-22T05:14:41.952 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:14:41 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: ignoring --setgroup ceph since I am not root 2024-03-22T05:14:42.452 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:14:42 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:14:42.097+0000 7facd74c4000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-22T05:14:42.452 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:14:42 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:14:42.188+0000 7facd74c4000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-22T05:14:42.883 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:42 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/2506956309' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-22T05:14:42.883 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:42 smithi005 ceph-mon[24793]: mgrmap e5: y(active, since 10s) 2024-03-22T05:14:42.883 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:14:42 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:14:42.793+0000 7facd74c4000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-22T05:14:43.198 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:14:42 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:14:42.883+0000 7facd74c4000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-22T05:14:43.198 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:14:42 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:14:42.967+0000 7facd74c4000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-22T05:14:43.198 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:14:43 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:14:43.117+0000 7facd74c4000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-22T05:14:43.199 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:14:43 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:14:43.198+0000 7facd74c4000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-22T05:14:43.452 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:14:43 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:14:43.311+0000 7facd74c4000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-22T05:14:43.452 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:14:43 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:14:43.396+0000 7facd74c4000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-22T05:14:43.952 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:14:43 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:14:43.560+0000 7facd74c4000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-22T05:14:43.952 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:14:43 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:14:43.646+0000 7facd74c4000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-22T05:14:44.823 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:14:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:14:44.701+0000 7facd74c4000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-22T05:14:44.823 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: { 2024-03-22T05:14:44.823 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "epoch": 5, 2024-03-22T05:14:44.824 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "available": true, 2024-03-22T05:14:44.824 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "active_name": "y", 2024-03-22T05:14:44.824 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "num_standby": 0 2024-03-22T05:14:44.824 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: } 2024-03-22T05:14:45.119 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:44 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/3144132004' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-22T05:14:45.119 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:14:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:14:44.834+0000 7facd74c4000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-22T05:14:45.119 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:14:45 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:14:45.030+0000 7facd74c4000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-22T05:14:45.119 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:14:45 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:14:45.118+0000 7facd74c4000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-22T05:14:45.447 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:14:45 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:14:45.257+0000 7facd74c4000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-22T05:14:45.451 INFO:teuthology.orchestra.run.smithi005.stderr:Waiting for the mgr to restart... 2024-03-22T05:14:45.451 INFO:teuthology.orchestra.run.smithi005.stderr:Waiting for mgr epoch 5... 2024-03-22T05:14:45.702 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:14:45 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:14:45.585+0000 7facd74c4000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-22T05:14:46.075 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:14:45 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:14:45.815+0000 7facd74c4000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-22T05:14:46.076 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:14:45 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:14:45.903+0000 7facd74c4000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-22T05:14:46.076 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:14:46 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:14:46.076+0000 7facd74c4000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-22T05:14:47.309 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:14:47 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:14:47.008+0000 7facd74c4000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-22T05:14:47.309 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:14:47 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:14:47.309+0000 7facd74c4000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-22T05:14:47.702 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:14:47 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:14:47.405+0000 7facd74c4000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-22T05:14:47.702 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:14:47 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:14:47.558+0000 7facd74c4000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-22T05:14:48.632 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: { 2024-03-22T05:14:48.633 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-03-22T05:14:48.633 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "initialized": true 2024-03-22T05:14:48.633 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: } 2024-03-22T05:14:48.946 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:48 smithi005 ceph-mon[24793]: Active manager daemon y restarted 2024-03-22T05:14:48.946 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:48 smithi005 ceph-mon[24793]: Activating manager daemon y 2024-03-22T05:14:48.946 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:48 smithi005 ceph-mon[24793]: osdmap e2: 0 total, 0 up, 0 in 2024-03-22T05:14:48.946 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:48 smithi005 ceph-mon[24793]: mgrmap e6: y(active, starting, since 0.0526188s) 2024-03-22T05:14:48.946 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:48 smithi005 ceph-mon[24793]: from='mgr.14118 172.21.15.5:0/209254006' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-22T05:14:48.946 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:48 smithi005 ceph-mon[24793]: from='mgr.14118 172.21.15.5:0/209254006' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-22T05:14:48.946 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:48 smithi005 ceph-mon[24793]: from='mgr.14118 172.21.15.5:0/209254006' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-22T05:14:48.947 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:48 smithi005 ceph-mon[24793]: from='mgr.14118 172.21.15.5:0/209254006' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-22T05:14:48.947 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:48 smithi005 ceph-mon[24793]: from='mgr.14118 172.21.15.5:0/209254006' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-22T05:14:48.947 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:48 smithi005 ceph-mon[24793]: Manager daemon y is now available 2024-03-22T05:14:48.947 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:48 smithi005 ceph-mon[24793]: from='mgr.14118 172.21.15.5:0/209254006' entity='mgr.y' 2024-03-22T05:14:48.947 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:48 smithi005 ceph-mon[24793]: from='mgr.14118 172.21.15.5:0/209254006' entity='mgr.y' 2024-03-22T05:14:48.947 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:48 smithi005 ceph-mon[24793]: from='mgr.14118 172.21.15.5:0/209254006' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:14:48.947 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:48 smithi005 ceph-mon[24793]: from='mgr.14118 172.21.15.5:0/209254006' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:14:48.947 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:48 smithi005 ceph-mon[24793]: from='mgr.14118 172.21.15.5:0/209254006' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:14:48.947 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:48 smithi005 ceph-mon[24793]: from='mgr.14118 172.21.15.5:0/209254006' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-22T05:14:48.947 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:48 smithi005 ceph-mon[24793]: from='mgr.14118 172.21.15.5:0/209254006' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-22T05:14:49.239 INFO:teuthology.orchestra.run.smithi005.stderr:mgr epoch 5 is available 2024-03-22T05:14:49.239 INFO:teuthology.orchestra.run.smithi005.stderr:Setting orchestrator backend to cephadm... 2024-03-22T05:14:49.829 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:49 smithi005 ceph-mon[24793]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-22T05:14:49.829 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:49 smithi005 ceph-mon[24793]: mgrmap e7: y(active, since 1.05459s) 2024-03-22T05:14:49.829 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:49 smithi005 ceph-mon[24793]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-22T05:14:50.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:50 smithi005 ceph-mon[24793]: mgrmap e8: y(active, since 2s) 2024-03-22T05:14:50.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:50 smithi005 ceph-mon[24793]: from='mgr.14118 172.21.15.5:0/209254006' entity='mgr.y' 2024-03-22T05:14:50.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:50 smithi005 ceph-mon[24793]: from='mgr.14118 172.21.15.5:0/209254006' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:14:50.702 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:14:50 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: [22/Mar/2024:05:14:50] ENGINE Bus STARTING 2024-03-22T05:14:50.924 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: value unchanged 2024-03-22T05:14:51.157 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:14:50 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: [22/Mar/2024:05:14:50] ENGINE Serving on https://172.21.15.5:7150 2024-03-22T05:14:51.157 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:14:50 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: [22/Mar/2024:05:14:50] ENGINE Bus STARTED 2024-03-22T05:14:51.393 INFO:teuthology.orchestra.run.smithi005.stderr:Generating ssh key... 2024-03-22T05:14:51.786 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:51 smithi005 ceph-mon[24793]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:14:51.786 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:51 smithi005 ceph-mon[24793]: from='mgr.14118 172.21.15.5:0/209254006' entity='mgr.y' 2024-03-22T05:14:51.786 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:51 smithi005 ceph-mon[24793]: from='mgr.14118 172.21.15.5:0/209254006' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:14:52.605 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:14:52 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: Generating public/private rsa key pair. 2024-03-22T05:14:52.605 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:14:52 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: Your identification has been saved in /tmp/tmp7wq67qt0/key. 2024-03-22T05:14:52.606 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:14:52 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: Your public key has been saved in /tmp/tmp7wq67qt0/key.pub. 2024-03-22T05:14:52.606 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:14:52 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: The key fingerprint is: 2024-03-22T05:14:52.606 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:14:52 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: SHA256:QtPy2EcObbgD9vvt5uyVvPLUkRf9e0RBT4n+xzlW/0g ceph-f59f36e2-e80a-11ee-95cd-87774f69a715 2024-03-22T05:14:52.606 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:14:52 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: The key's randomart image is: 2024-03-22T05:14:52.606 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:14:52 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: +---[RSA 3072]----+ 2024-03-22T05:14:52.606 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:14:52 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: | ooo| 2024-03-22T05:14:52.606 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:14:52 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: | . o . o+| 2024-03-22T05:14:52.606 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:14:52 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: | * + + . .+| 2024-03-22T05:14:52.606 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:14:52 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: | o O * . .=| 2024-03-22T05:14:52.606 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:14:52 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: | o S o .+B| 2024-03-22T05:14:52.606 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:14:52 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: | . + .EOB| 2024-03-22T05:14:52.606 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:14:52 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: | . .*o*| 2024-03-22T05:14:52.606 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:14:52 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: | . ooo..o| 2024-03-22T05:14:52.606 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:14:52 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: | .=*oo | 2024-03-22T05:14:52.607 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:14:52 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: +----[SHA256]-----+ 2024-03-22T05:14:52.915 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:52 smithi005 ceph-mon[24793]: [22/Mar/2024:05:14:50] ENGINE Bus STARTING 2024-03-22T05:14:52.915 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:52 smithi005 ceph-mon[24793]: [22/Mar/2024:05:14:50] ENGINE Serving on https://172.21.15.5:7150 2024-03-22T05:14:52.915 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:52 smithi005 ceph-mon[24793]: [22/Mar/2024:05:14:50] ENGINE Bus STARTED 2024-03-22T05:14:52.915 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:52 smithi005 ceph-mon[24793]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:14:52.915 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:52 smithi005 ceph-mon[24793]: from='mgr.14118 172.21.15.5:0/209254006' entity='mgr.y' 2024-03-22T05:14:52.915 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:52 smithi005 ceph-mon[24793]: from='mgr.14118 172.21.15.5:0/209254006' entity='mgr.y' 2024-03-22T05:14:53.420 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDGdSCaYFe9GP4kujr6cXRr0EK2geMtfPoUs3tWSfoJd65B5uhVX7a86981tWOSBmKw9rQEMaY+e1OOXTpE0CCSJfG0u/k4pWtjeinybpyYYnuGpI6374frNx7uKPgddMbInG41Lc8QalMvkL512rhFdY1eUgyCVr47Nm9M16fE/hw6dkfsyIC5o7J6mT13j9C8IfLecKwV28pAsAJw5Sny/b0caOm3bLXSbB2MrG+tBZfb2moqxW0MMCmMdFJI2XqSETOZvW/x+MeMZqkuMzcFoI5QbLZZTBgcj6NUBoptwB0uMEn1nCmNGSNTD05O9Yp8CzKXCiLCZl6SugqOJGkX/WktOvcTx5OFG+t18tUyXQZ8AjfEwxOAV19/YeMjonifKypu2FgHSQeBbXGMGOmwFszXctjY1sykiv7SFaIwAFjqENwamOwU/1nCmBXXlyVrBJ0jkKOARQBNyCa63K6wMS7fOUjzkQP/egtwuUAuWGJPWFCPEy9gy6STKTgqkt8= ceph-f59f36e2-e80a-11ee-95cd-87774f69a715 2024-03-22T05:14:53.917 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:53 smithi005 ceph-mon[24793]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:14:53.917 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:53 smithi005 ceph-mon[24793]: Generating ssh key... 2024-03-22T05:14:53.940 INFO:teuthology.orchestra.run.smithi005.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-22T05:14:53.940 INFO:teuthology.orchestra.run.smithi005.stderr:Adding key to root@localhost authorized_keys... 2024-03-22T05:14:53.941 INFO:teuthology.orchestra.run.smithi005.stderr:Adding host smithi005... 2024-03-22T05:14:54.840 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:54 smithi005 ceph-mon[24793]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:14:55.533 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: Added host 'smithi005' with addr '172.21.15.5' 2024-03-22T05:14:55.844 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:55 smithi005 ceph-mon[24793]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi005", "addr": "172.21.15.5", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:14:55.845 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:55 smithi005 ceph-mon[24793]: from='mgr.14118 172.21.15.5:0/209254006' entity='mgr.y' 2024-03-22T05:14:55.845 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:55 smithi005 ceph-mon[24793]: from='mgr.14118 172.21.15.5:0/209254006' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:14:56.114 INFO:teuthology.orchestra.run.smithi005.stderr:Deploying unmanaged mon service... 2024-03-22T05:14:56.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:56 smithi005 ceph-mon[24793]: Deploying cephadm binary to smithi005 2024-03-22T05:14:56.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:56 smithi005 ceph-mon[24793]: Added host smithi005 2024-03-22T05:14:56.710 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: Scheduled mon update... 2024-03-22T05:14:57.426 INFO:teuthology.orchestra.run.smithi005.stderr:Deploying unmanaged mgr service... 2024-03-22T05:14:57.823 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:57 smithi005 ceph-mon[24793]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:14:57.823 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:57 smithi005 ceph-mon[24793]: Saving service mon spec with placement count:5 2024-03-22T05:14:57.824 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:57 smithi005 ceph-mon[24793]: from='mgr.14118 172.21.15.5:0/209254006' entity='mgr.y' 2024-03-22T05:14:58.062 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-03-22T05:14:59.233 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:59 smithi005 ceph-mon[24793]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:14:59.233 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:59 smithi005 ceph-mon[24793]: Saving service mgr spec with placement count:2 2024-03-22T05:14:59.233 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:59 smithi005 ceph-mon[24793]: from='mgr.14118 172.21.15.5:0/209254006' entity='mgr.y' 2024-03-22T05:14:59.233 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:14:59 smithi005 ceph-mon[24793]: from='mgr.14118 172.21.15.5:0/209254006' entity='mgr.y' 2024-03-22T05:15:00.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:00 smithi005 ceph-mon[24793]: from='mgr.14118 172.21.15.5:0/209254006' entity='mgr.y' 2024-03-22T05:15:00.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:00 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/386890611' entity='client.admin' 2024-03-22T05:15:02.259 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:02 smithi005 ceph-mon[24793]: from='mgr.14118 172.21.15.5:0/209254006' entity='mgr.y' 2024-03-22T05:15:02.259 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:02 smithi005 ceph-mon[24793]: from='mgr.14118 172.21.15.5:0/209254006' entity='mgr.y' 2024-03-22T05:15:02.259 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:02 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/2044129463' entity='client.admin' 2024-03-22T05:15:02.261 INFO:teuthology.orchestra.run.smithi005.stderr:Enabling the dashboard module... 2024-03-22T05:15:03.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:03 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/1208623547' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-22T05:15:03.202 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:15:03 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: ignoring --setuser ceph since I am not root 2024-03-22T05:15:03.202 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:15:03 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: ignoring --setgroup ceph since I am not root 2024-03-22T05:15:03.810 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:15:03 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:15:03.556+0000 7f1417018000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-22T05:15:03.810 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:15:03 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:15:03.647+0000 7f1417018000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-22T05:15:04.077 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:04 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/1208623547' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-22T05:15:04.077 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:04 smithi005 ceph-mon[24793]: mgrmap e9: y(active, since 15s) 2024-03-22T05:15:04.353 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:15:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:15:04.258+0000 7f1417018000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-22T05:15:04.354 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:15:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:15:04.352+0000 7f1417018000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-22T05:15:04.624 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:15:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:15:04.440+0000 7f1417018000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-22T05:15:04.624 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:15:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:15:04.595+0000 7f1417018000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-22T05:15:04.734 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: { 2024-03-22T05:15:04.734 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "epoch": 9, 2024-03-22T05:15:04.734 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "available": true, 2024-03-22T05:15:04.734 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "active_name": "y", 2024-03-22T05:15:04.734 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "num_standby": 0 2024-03-22T05:15:04.734 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: } 2024-03-22T05:15:04.879 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:15:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:15:04.678+0000 7f1417018000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-22T05:15:04.879 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:15:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:15:04.793+0000 7f1417018000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-22T05:15:04.879 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:15:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:15:04.878+0000 7f1417018000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-22T05:15:05.131 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:05 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/352422741' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-22T05:15:05.131 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:15:05 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:15:05.043+0000 7f1417018000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-22T05:15:05.452 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:15:05 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:15:05.130+0000 7f1417018000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-22T05:15:05.609 INFO:teuthology.orchestra.run.smithi005.stderr:Waiting for the mgr to restart... 2024-03-22T05:15:05.609 INFO:teuthology.orchestra.run.smithi005.stderr:Waiting for mgr epoch 9... 2024-03-22T05:15:06.452 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:15:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:15:06.189+0000 7f1417018000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-22T05:15:06.452 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:15:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:15:06.322+0000 7f1417018000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-22T05:15:06.952 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:15:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:15:06.516+0000 7f1417018000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-22T05:15:06.952 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:15:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:15:06.605+0000 7f1417018000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-22T05:15:06.952 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:15:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:15:06.744+0000 7f1417018000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-22T05:15:07.380 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:15:07 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:15:07.066+0000 7f1417018000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-22T05:15:07.380 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:15:07 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:15:07.292+0000 7f1417018000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-22T05:15:07.380 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:15:07 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:15:07.379+0000 7f1417018000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-22T05:15:07.702 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:15:07 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:15:07.558+0000 7f1417018000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-22T05:15:08.821 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:15:08 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:15:08.514+0000 7f1417018000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-22T05:15:08.821 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:15:08 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:15:08.820+0000 7f1417018000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-22T05:15:09.072 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:15:08 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:15:08.917+0000 7f1417018000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-22T05:15:09.072 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:15:09 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:15:09.071+0000 7f1417018000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-22T05:15:10.059 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:15:09 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: [22/Mar/2024:05:15:09] ENGINE Bus STARTING 2024-03-22T05:15:10.059 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:15:09 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: [22/Mar/2024:05:15:09] ENGINE Serving on https://172.21.15.5:7150 2024-03-22T05:15:10.059 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:15:09 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: [22/Mar/2024:05:15:09] ENGINE Bus STARTED 2024-03-22T05:15:10.147 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: { 2024-03-22T05:15:10.147 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2024-03-22T05:15:10.147 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: "initialized": true 2024-03-22T05:15:10.147 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: } 2024-03-22T05:15:10.386 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:10 smithi005 ceph-mon[24793]: Active manager daemon y restarted 2024-03-22T05:15:10.386 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:10 smithi005 ceph-mon[24793]: Activating manager daemon y 2024-03-22T05:15:10.386 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:10 smithi005 ceph-mon[24793]: osdmap e3: 0 total, 0 up, 0 in 2024-03-22T05:15:10.386 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:10 smithi005 ceph-mon[24793]: mgrmap e10: y(active, starting, since 0.0524177s) 2024-03-22T05:15:10.387 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:10 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-22T05:15:10.387 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:10 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-22T05:15:10.387 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:10 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-22T05:15:10.387 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:10 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-22T05:15:10.387 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:10 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-22T05:15:10.387 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:10 smithi005 ceph-mon[24793]: Manager daemon y is now available 2024-03-22T05:15:10.387 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:10 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:15:10.387 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:10 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:15:10.387 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:10 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-22T05:15:10.387 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:10 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-22T05:15:10.388 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:10 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:15:10.991 INFO:teuthology.orchestra.run.smithi005.stderr:mgr epoch 9 is available 2024-03-22T05:15:10.991 INFO:teuthology.orchestra.run.smithi005.stderr:Generating a dashboard self-signed certificate... 2024-03-22T05:15:11.242 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:11 smithi005 ceph-mon[24793]: [22/Mar/2024:05:15:09] ENGINE Bus STARTING 2024-03-22T05:15:11.243 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:11 smithi005 ceph-mon[24793]: [22/Mar/2024:05:15:09] ENGINE Serving on https://172.21.15.5:7150 2024-03-22T05:15:11.243 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:11 smithi005 ceph-mon[24793]: [22/Mar/2024:05:15:09] ENGINE Bus STARTED 2024-03-22T05:15:11.243 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:11 smithi005 ceph-mon[24793]: mgrmap e11: y(active, since 1.05509s) 2024-03-22T05:15:11.695 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: Self-signed certificate created 2024-03-22T05:15:12.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:12 smithi005 ceph-mon[24793]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-22T05:15:12.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:12 smithi005 ceph-mon[24793]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-22T05:15:12.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:12 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:15:12.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:12 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:15:12.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:12 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:15:12.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:12 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:15:12.407 INFO:teuthology.orchestra.run.smithi005.stderr:Creating initial admin user... 2024-03-22T05:15:13.267 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$CwairoQsuL8hc5duE3nZ4O/jZ/oJqSWKoCBSL84bnJMWRXIURVBOy", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1711084513, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-22T05:15:13.352 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:13 smithi005 ceph-mon[24793]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:15:13.353 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:13 smithi005 ceph-mon[24793]: mgrmap e12: y(active, since 3s) 2024-03-22T05:15:14.157 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:14 smithi005 ceph-mon[24793]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:15:14.157 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:14 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:15:14.737 INFO:teuthology.orchestra.run.smithi005.stderr:Fetching dashboard port number... 2024-03-22T05:15:15.390 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: 8443 2024-03-22T05:15:16.005 INFO:teuthology.orchestra.run.smithi005.stderr:systemctl: disabled 2024-03-22T05:15:16.013 INFO:teuthology.orchestra.run.smithi005.stderr:systemctl: inactive 2024-03-22T05:15:16.014 INFO:teuthology.orchestra.run.smithi005.stderr:firewalld.service is not enabled 2024-03-22T05:15:16.014 INFO:teuthology.orchestra.run.smithi005.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-22T05:15:16.016 INFO:teuthology.orchestra.run.smithi005.stderr:Ceph Dashboard is now available at: 2024-03-22T05:15:16.016 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-03-22T05:15:16.016 INFO:teuthology.orchestra.run.smithi005.stderr: URL: https://smithi005.front.sepia.ceph.com:8443/ 2024-03-22T05:15:16.016 INFO:teuthology.orchestra.run.smithi005.stderr: User: admin 2024-03-22T05:15:16.016 INFO:teuthology.orchestra.run.smithi005.stderr: Password: bxka2wqb5g 2024-03-22T05:15:16.016 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-03-22T05:15:16.016 INFO:teuthology.orchestra.run.smithi005.stderr:Enabling autotune for osd_memory_target 2024-03-22T05:15:16.390 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:16 smithi005 ceph-mon[24793]: mgrmap e13: y(active, since 6s) 2024-03-22T05:15:16.391 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:16 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/1814038360' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-22T05:15:17.952 INFO:teuthology.orchestra.run.smithi005.stderr:/usr/bin/ceph: set mgr/dashboard/cluster/status 2024-03-22T05:15:18.603 INFO:teuthology.orchestra.run.smithi005.stderr:You can access the Ceph CLI with: 2024-03-22T05:15:18.669 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-03-22T05:15:18.669 INFO:teuthology.orchestra.run.smithi005.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-22T05:15:18.669 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-03-22T05:15:18.669 INFO:teuthology.orchestra.run.smithi005.stderr:Please consider enabling telemetry to help improve Ceph: 2024-03-22T05:15:18.669 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-03-22T05:15:18.669 INFO:teuthology.orchestra.run.smithi005.stderr: ceph telemetry on 2024-03-22T05:15:18.669 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-03-22T05:15:18.669 INFO:teuthology.orchestra.run.smithi005.stderr:For more information see: 2024-03-22T05:15:18.669 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-03-22T05:15:18.670 INFO:teuthology.orchestra.run.smithi005.stderr: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-03-22T05:15:18.670 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-03-22T05:15:18.670 INFO:teuthology.orchestra.run.smithi005.stderr:Bootstrap complete. 2024-03-22T05:15:18.671 INFO:tasks.cephadm:Fetching config... 2024-03-22T05:15:18.672 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-03-22T05:15:18.672 DEBUG:teuthology.orchestra.run.smithi005:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-22T05:15:18.687 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-22T05:15:18.687 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-03-22T05:15:18.687 DEBUG:teuthology.orchestra.run.smithi005:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-22T05:15:18.742 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-22T05:15:18.743 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-03-22T05:15:18.743 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-22T05:15:18.811 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-22T05:15:18.811 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-03-22T05:15:18.811 DEBUG:teuthology.orchestra.run.smithi005:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-22T05:15:18.869 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-22T05:15:18.869 DEBUG:teuthology.orchestra.run.smithi005:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDGdSCaYFe9GP4kujr6cXRr0EK2geMtfPoUs3tWSfoJd65B5uhVX7a86981tWOSBmKw9rQEMaY+e1OOXTpE0CCSJfG0u/k4pWtjeinybpyYYnuGpI6374frNx7uKPgddMbInG41Lc8QalMvkL512rhFdY1eUgyCVr47Nm9M16fE/hw6dkfsyIC5o7J6mT13j9C8IfLecKwV28pAsAJw5Sny/b0caOm3bLXSbB2MrG+tBZfb2moqxW0MMCmMdFJI2XqSETOZvW/x+MeMZqkuMzcFoI5QbLZZTBgcj6NUBoptwB0uMEn1nCmNGSNTD05O9Yp8CzKXCiLCZl6SugqOJGkX/WktOvcTx5OFG+t18tUyXQZ8AjfEwxOAV19/YeMjonifKypu2FgHSQeBbXGMGOmwFszXctjY1sykiv7SFaIwAFjqENwamOwU/1nCmBXXlyVrBJ0jkKOARQBNyCa63K6wMS7fOUjzkQP/egtwuUAuWGJPWFCPEy9gy6STKTgqkt8= ceph-f59f36e2-e80a-11ee-95cd-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-22T05:15:18.951 INFO:teuthology.orchestra.run.smithi005.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDGdSCaYFe9GP4kujr6cXRr0EK2geMtfPoUs3tWSfoJd65B5uhVX7a86981tWOSBmKw9rQEMaY+e1OOXTpE0CCSJfG0u/k4pWtjeinybpyYYnuGpI6374frNx7uKPgddMbInG41Lc8QalMvkL512rhFdY1eUgyCVr47Nm9M16fE/hw6dkfsyIC5o7J6mT13j9C8IfLecKwV28pAsAJw5Sny/b0caOm3bLXSbB2MrG+tBZfb2moqxW0MMCmMdFJI2XqSETOZvW/x+MeMZqkuMzcFoI5QbLZZTBgcj6NUBoptwB0uMEn1nCmNGSNTD05O9Yp8CzKXCiLCZl6SugqOJGkX/WktOvcTx5OFG+t18tUyXQZ8AjfEwxOAV19/YeMjonifKypu2FgHSQeBbXGMGOmwFszXctjY1sykiv7SFaIwAFjqENwamOwU/1nCmBXXlyVrBJ0jkKOARQBNyCa63K6wMS7fOUjzkQP/egtwuUAuWGJPWFCPEy9gy6STKTgqkt8= ceph-f59f36e2-e80a-11ee-95cd-87774f69a715 2024-03-22T05:15:18.965 DEBUG:teuthology.orchestra.run.smithi186:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDGdSCaYFe9GP4kujr6cXRr0EK2geMtfPoUs3tWSfoJd65B5uhVX7a86981tWOSBmKw9rQEMaY+e1OOXTpE0CCSJfG0u/k4pWtjeinybpyYYnuGpI6374frNx7uKPgddMbInG41Lc8QalMvkL512rhFdY1eUgyCVr47Nm9M16fE/hw6dkfsyIC5o7J6mT13j9C8IfLecKwV28pAsAJw5Sny/b0caOm3bLXSbB2MrG+tBZfb2moqxW0MMCmMdFJI2XqSETOZvW/x+MeMZqkuMzcFoI5QbLZZTBgcj6NUBoptwB0uMEn1nCmNGSNTD05O9Yp8CzKXCiLCZl6SugqOJGkX/WktOvcTx5OFG+t18tUyXQZ8AjfEwxOAV19/YeMjonifKypu2FgHSQeBbXGMGOmwFszXctjY1sykiv7SFaIwAFjqENwamOwU/1nCmBXXlyVrBJ0jkKOARQBNyCa63K6wMS7fOUjzkQP/egtwuUAuWGJPWFCPEy9gy6STKTgqkt8= ceph-f59f36e2-e80a-11ee-95cd-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-22T05:15:19.011 INFO:teuthology.orchestra.run.smithi186.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDGdSCaYFe9GP4kujr6cXRr0EK2geMtfPoUs3tWSfoJd65B5uhVX7a86981tWOSBmKw9rQEMaY+e1OOXTpE0CCSJfG0u/k4pWtjeinybpyYYnuGpI6374frNx7uKPgddMbInG41Lc8QalMvkL512rhFdY1eUgyCVr47Nm9M16fE/hw6dkfsyIC5o7J6mT13j9C8IfLecKwV28pAsAJw5Sny/b0caOm3bLXSbB2MrG+tBZfb2moqxW0MMCmMdFJI2XqSETOZvW/x+MeMZqkuMzcFoI5QbLZZTBgcj6NUBoptwB0uMEn1nCmNGSNTD05O9Yp8CzKXCiLCZl6SugqOJGkX/WktOvcTx5OFG+t18tUyXQZ8AjfEwxOAV19/YeMjonifKypu2FgHSQeBbXGMGOmwFszXctjY1sykiv7SFaIwAFjqENwamOwU/1nCmBXXlyVrBJ0jkKOARQBNyCa63K6wMS7fOUjzkQP/egtwuUAuWGJPWFCPEy9gy6STKTgqkt8= ceph-f59f36e2-e80a-11ee-95cd-87774f69a715 2024-03-22T05:15:19.027 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-22T05:15:19.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:18 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/1669882662' entity='client.admin' 2024-03-22T05:15:22.626 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-22T05:15:22.626 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-22T05:15:23.251 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:22 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/1343267424' entity='client.admin' 2024-03-22T05:15:24.136 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:24 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:15:24.137 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:24 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi005", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:15:24.137 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:24 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:15:24.137 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:24 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:15:24.137 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:24 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:15:25.188 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi186 2024-03-22T05:15:25.188 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-03-22T05:15:25.188 DEBUG:teuthology.orchestra.run.smithi186:> dd of=/etc/ceph/ceph.conf 2024-03-22T05:15:25.212 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-03-22T05:15:25.212 DEBUG:teuthology.orchestra.run.smithi186:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-22T05:15:25.270 INFO:tasks.cephadm:Adding host smithi186 to orchestrator... 2024-03-22T05:15:25.270 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -- ceph orch host add smithi186 2024-03-22T05:15:25.836 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:25 smithi005 ceph-mon[24793]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:15:25.836 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:25 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:15:25.836 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:25 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:15:25.836 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:25 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:15:25.836 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:25 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:15:25.836 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:25 smithi005 ceph-mon[24793]: Updating smithi005:/etc/ceph/ceph.conf 2024-03-22T05:15:25.836 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:25 smithi005 ceph-mon[24793]: Updating smithi005:/etc/ceph/ceph.client.admin.keyring 2024-03-22T05:15:25.836 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:25 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:15:25.836 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:25 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:15:27.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:27 smithi005 ceph-mon[24793]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi186", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:15:28.073 INFO:teuthology.orchestra.run.smithi005.stdout:Added host 'smithi186' with addr '172.21.15.186' 2024-03-22T05:15:28.712 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -- ceph orch host ls --format=json 2024-03-22T05:15:29.348 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:29 smithi005 ceph-mon[24793]: Deploying cephadm binary to smithi186 2024-03-22T05:15:29.349 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:29 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:15:29.349 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:29 smithi005 ceph-mon[24793]: Added host smithi186 2024-03-22T05:15:29.349 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:29 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:15:29.349 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:29 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:15:29.349 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:29 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:15:29.349 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:29 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:15:29.349 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:29 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:15:30.605 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-22T05:15:30.605 INFO:teuthology.orchestra.run.smithi005.stdout:[{"addr": "172.21.15.5", "hostname": "smithi005", "labels": [], "status": ""}, {"addr": "172.21.15.186", "hostname": "smithi186", "labels": [], "status": ""}] 2024-03-22T05:15:30.899 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:30 smithi005 ceph-mon[24793]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T05:15:30.900 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:30 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:15:31.166 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-22T05:15:31.166 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -- ceph osd crush tunables default 2024-03-22T05:15:31.787 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:31 smithi005 ceph-mon[24793]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-22T05:15:32.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:32 smithi005 ceph-mon[24793]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T05:15:32.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:32 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:15:32.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:32 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:15:32.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:32 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi186", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:15:32.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:32 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:15:33.719 INFO:teuthology.orchestra.run.smithi005.stderr:adjusted tunables profile to default 2024-03-22T05:15:34.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:33 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/103743364' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-22T05:15:34.465 INFO:tasks.cephadm:Adding mon.a on smithi005 2024-03-22T05:15:34.465 INFO:tasks.cephadm:Adding mon.c on smithi005 2024-03-22T05:15:34.465 INFO:tasks.cephadm:Adding mon.b on smithi186 2024-03-22T05:15:34.465 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -- ceph orch apply mon '3;smithi005:172.21.15.5=a;smithi005:[v2:172.21.15.5:3301,v1:172.21.15.5:6790]=c;smithi186:172.21.15.186=b' 2024-03-22T05:15:35.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:34 smithi005 ceph-mon[24793]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T05:15:35.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:34 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/103743364' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-22T05:15:35.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:34 smithi005 ceph-mon[24793]: osdmap e4: 0 total, 0 up, 0 in 2024-03-22T05:15:36.202 INFO:teuthology.orchestra.run.smithi186.stdout:Scheduled mon update... 2024-03-22T05:15:36.881 DEBUG:teuthology.orchestra.run.smithi005:mon.c> sudo journalctl -f -n 0 -u ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@mon.c.service 2024-03-22T05:15:36.884 DEBUG:teuthology.orchestra.run.smithi186:mon.b> sudo journalctl -f -n 0 -u ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@mon.b.service 2024-03-22T05:15:36.886 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-22T05:15:36.886 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-22T05:15:37.004 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:36 smithi005 ceph-mon[24793]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T05:15:37.004 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:36 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:15:37.004 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:36 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:15:37.004 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:36 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:15:37.004 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:36 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:15:37.860 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:37 smithi005 ceph-mon[24793]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi005:172.21.15.5=a;smithi005:[v2:172.21.15.5:3301,v1:172.21.15.5:6790]=c;smithi186:172.21.15.186=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:15:37.861 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:37 smithi005 ceph-mon[24793]: Saving service mon spec with placement smithi005:172.21.15.5=a;smithi005:[v2:172.21.15.5:3301,v1:172.21.15.5:6790]=c;smithi186:172.21.15.186=b;count:3 2024-03-22T05:15:37.861 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:37 smithi005 ceph-mon[24793]: Updating smithi186:/etc/ceph/ceph.conf 2024-03-22T05:15:37.861 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:37 smithi005 ceph-mon[24793]: Updating smithi186:/etc/ceph/ceph.client.admin.keyring 2024-03-22T05:15:37.861 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:37 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:15:37.861 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:37 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:15:37.861 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:37 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-22T05:15:37.861 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:37 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:15:37.861 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:37 smithi005 ceph-mon[24793]: Deploying daemon mon.c on smithi005 2024-03-22T05:15:38.662 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-22T05:15:38.664 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":1,"fsid":"f59f36e2-e80a-11ee-95cd-87774f69a715","modified":"2024-03-22T05:14:16.214419Z","created":"2024-03-22T05:14:16.214419Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-22T05:15:38.665 INFO:teuthology.orchestra.run.smithi186.stderr:dumped monmap epoch 1 2024-03-22T05:15:39.352 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[24793]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T05:15:39.353 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[24793]: from='client.? 172.21.15.186:0/1576193235' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-22T05:15:39.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 systemd[1]: Starting Ceph mon.c for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:15:39.980 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 podman[29504]: 2024-03-22 05:15:39.728139911 +0000 UTC m=+0.113896400 container create 00ec24d38612c9e5b2bd914999a6c666bcc53331ecb8cfd79900c97c98e16c16 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mon-c, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, release=754, io.buildah.version=1.19.8, RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, version=8, name=centos-stream) 2024-03-22T05:15:39.980 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 podman[29504]: 2024-03-22 05:15:39.633034603 +0000 UTC m=+0.018791093 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-03-22T05:15:39.980 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 podman[29504]: 2024-03-22 05:15:39.897343339 +0000 UTC m=+0.283099828 container init 00ec24d38612c9e5b2bd914999a6c666bcc53331ecb8cfd79900c97c98e16c16 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mon-c, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, distribution-scope=public, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, version=8, ceph=True, name=centos-stream, com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-03-22T05:15:39.980 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 podman[29504]: 2024-03-22 05:15:39.900325342 +0000 UTC m=+0.286081832 container start 00ec24d38612c9e5b2bd914999a6c666bcc53331ecb8cfd79900c97c98e16c16 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mon-c, GIT_BRANCH=HEAD, io.openshift.expose-services=, vcs-type=git, version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, distribution-scope=public, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8) 2024-03-22T05:15:39.980 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: set uid:gid to 167:167 (ceph:ceph) 2024-03-22T05:15:39.981 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-03-22T05:15:39.981 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: pidfile_write: ignore empty --pid-file 2024-03-22T05:15:39.981 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: load: jerasure load: lrc 2024-03-22T05:15:39.981 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: RocksDB version: 6.15.5 2024-03-22T05:15:39.981 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-03-22T05:15:39.981 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Compile date Apr 18 2022 2024-03-22T05:15:39.981 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: DB SUMMARY 2024-03-22T05:15:39.981 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: DB Session ID: 6I64R0QV82F4NIN8RJTW 2024-03-22T05:15:39.981 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: CURRENT file: CURRENT 2024-03-22T05:15:39.981 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: IDENTITY file: IDENTITY 2024-03-22T05:15:39.981 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2024-03-22T05:15:39.981 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-03-22T05:15:39.981 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-03-22T05:15:39.982 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.error_if_exists: 0 2024-03-22T05:15:39.982 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.create_if_missing: 0 2024-03-22T05:15:39.982 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.paranoid_checks: 1 2024-03-22T05:15:39.982 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-22T05:15:39.982 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.env: 0x55cb97ac6860 2024-03-22T05:15:39.982 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.fs: Posix File System 2024-03-22T05:15:39.982 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.info_log: 0x55cb98ab9dc0 2024-03-22T05:15:39.982 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.max_file_opening_threads: 16 2024-03-22T05:15:39.982 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.statistics: (nil) 2024-03-22T05:15:39.982 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.use_fsync: 0 2024-03-22T05:15:39.984 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.max_log_file_size: 0 2024-03-22T05:15:39.984 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-22T05:15:39.984 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-22T05:15:39.985 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.keep_log_file_num: 1000 2024-03-22T05:15:39.985 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.recycle_log_file_num: 0 2024-03-22T05:15:39.985 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.allow_fallocate: 1 2024-03-22T05:15:39.985 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.allow_mmap_reads: 0 2024-03-22T05:15:39.985 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.allow_mmap_writes: 0 2024-03-22T05:15:39.985 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.use_direct_reads: 0 2024-03-22T05:15:39.985 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-22T05:15:39.985 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.create_missing_column_families: 0 2024-03-22T05:15:39.985 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.db_log_dir: 2024-03-22T05:15:39.985 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-c/store.db 2024-03-22T05:15:39.985 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-22T05:15:39.986 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-22T05:15:39.986 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-22T05:15:39.986 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-22T05:15:39.986 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-22T05:15:39.986 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-22T05:15:39.986 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.advise_random_on_open: 1 2024-03-22T05:15:39.986 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.db_write_buffer_size: 0 2024-03-22T05:15:39.986 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.write_buffer_manager: 0x55cb98baa2d0 2024-03-22T05:15:39.986 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-22T05:15:39.986 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-03-22T05:15:39.986 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-22T05:15:39.986 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-22T05:15:39.986 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.rate_limiter: (nil) 2024-03-22T05:15:39.986 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-22T05:15:39.987 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.wal_recovery_mode: 2 2024-03-22T05:15:39.987 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.enable_thread_tracking: 0 2024-03-22T05:15:39.987 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.enable_pipelined_write: 0 2024-03-22T05:15:39.987 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.unordered_write: 0 2024-03-22T05:15:39.987 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-22T05:15:39.987 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-22T05:15:39.987 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-22T05:15:39.987 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-22T05:15:39.987 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.row_cache: None 2024-03-22T05:15:39.987 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.wal_filter: None 2024-03-22T05:15:39.987 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-22T05:15:39.987 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.allow_ingest_behind: 0 2024-03-22T05:15:39.987 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.preserve_deletes: 0 2024-03-22T05:15:39.987 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.two_write_queues: 0 2024-03-22T05:15:39.987 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.manual_wal_flush: 0 2024-03-22T05:15:39.988 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.atomic_flush: 0 2024-03-22T05:15:39.988 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-22T05:15:39.988 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-22T05:15:39.988 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-22T05:15:39.988 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.log_readahead_size: 0 2024-03-22T05:15:39.988 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-22T05:15:39.988 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.best_efforts_recovery: 0 2024-03-22T05:15:39.988 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-22T05:15:39.988 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-22T05:15:39.988 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.allow_data_in_errors: 0 2024-03-22T05:15:39.988 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.db_host_id: __hostname__ 2024-03-22T05:15:39.988 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.max_background_jobs: 2 2024-03-22T05:15:39.988 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.max_background_compactions: -1 2024-03-22T05:15:39.988 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.max_subcompactions: 1 2024-03-22T05:15:39.988 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-22T05:15:39.989 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-22T05:15:39.989 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-22T05:15:39.989 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.max_total_wal_size: 0 2024-03-22T05:15:39.989 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-22T05:15:39.989 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-22T05:15:39.989 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-22T05:15:39.989 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-22T05:15:39.989 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.max_open_files: -1 2024-03-22T05:15:39.989 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.bytes_per_sync: 0 2024-03-22T05:15:39.989 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-22T05:15:39.989 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-22T05:15:39.989 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.compaction_readahead_size: 0 2024-03-22T05:15:39.989 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.max_background_flushes: -1 2024-03-22T05:15:39.990 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Compression algorithms supported: 2024-03-22T05:15:39.990 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-22T05:15:39.990 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: kZSTD supported: 0 2024-03-22T05:15:39.990 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: kXpressCompression supported: 0 2024-03-22T05:15:39.990 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: kLZ4HCCompression supported: 1 2024-03-22T05:15:39.990 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: kLZ4Compression supported: 1 2024-03-22T05:15:39.990 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: kBZip2Compression supported: 0 2024-03-22T05:15:39.990 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: kZlibCompression supported: 1 2024-03-22T05:15:39.990 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: kSnappyCompression supported: 1 2024-03-22T05:15:39.990 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-22T05:15:39.990 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000003 2024-03-22T05:15:39.990 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-03-22T05:15:39.990 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-22T05:15:39.990 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.merge_operator: 2024-03-22T05:15:39.990 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.compaction_filter: None 2024-03-22T05:15:39.991 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.compaction_filter_factory: None 2024-03-22T05:15:39.991 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.sst_partitioner_factory: None 2024-03-22T05:15:39.991 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-22T05:15:39.991 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-22T05:15:39.991 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55cb98a85d00) 2024-03-22T05:15:39.991 INFO:journalctl@ceph.mon.c.smithi005.stdout: cache_index_and_filter_blocks: 1 2024-03-22T05:15:39.991 INFO:journalctl@ceph.mon.c.smithi005.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-22T05:15:39.991 INFO:journalctl@ceph.mon.c.smithi005.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-22T05:15:39.991 INFO:journalctl@ceph.mon.c.smithi005.stdout: pin_top_level_index_and_filter: 1 2024-03-22T05:15:39.991 INFO:journalctl@ceph.mon.c.smithi005.stdout: index_type: 0 2024-03-22T05:15:39.991 INFO:journalctl@ceph.mon.c.smithi005.stdout: data_block_index_type: 0 2024-03-22T05:15:39.991 INFO:journalctl@ceph.mon.c.smithi005.stdout: index_shortening: 1 2024-03-22T05:15:39.991 INFO:journalctl@ceph.mon.c.smithi005.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-22T05:15:39.991 INFO:journalctl@ceph.mon.c.smithi005.stdout: hash_index_allow_collision: 1 2024-03-22T05:15:39.992 INFO:journalctl@ceph.mon.c.smithi005.stdout: checksum: 1 2024-03-22T05:15:39.992 INFO:journalctl@ceph.mon.c.smithi005.stdout: no_block_cache: 0 2024-03-22T05:15:39.992 INFO:journalctl@ceph.mon.c.smithi005.stdout: block_cache: 0x55cb98af0170 2024-03-22T05:15:39.992 INFO:journalctl@ceph.mon.c.smithi005.stdout: block_cache_name: BinnedLRUCache 2024-03-22T05:15:39.992 INFO:journalctl@ceph.mon.c.smithi005.stdout: block_cache_options: 2024-03-22T05:15:39.992 INFO:journalctl@ceph.mon.c.smithi005.stdout: capacity : 536870912 2024-03-22T05:15:39.992 INFO:journalctl@ceph.mon.c.smithi005.stdout: num_shard_bits : 4 2024-03-22T05:15:39.992 INFO:journalctl@ceph.mon.c.smithi005.stdout: strict_capacity_limit : 0 2024-03-22T05:15:39.992 INFO:journalctl@ceph.mon.c.smithi005.stdout: high_pri_pool_ratio: 0.000 2024-03-22T05:15:39.992 INFO:journalctl@ceph.mon.c.smithi005.stdout: block_cache_compressed: (nil) 2024-03-22T05:15:39.992 INFO:journalctl@ceph.mon.c.smithi005.stdout: persistent_cache: (nil) 2024-03-22T05:15:39.992 INFO:journalctl@ceph.mon.c.smithi005.stdout: block_size: 4096 2024-03-22T05:15:39.992 INFO:journalctl@ceph.mon.c.smithi005.stdout: block_size_deviation: 10 2024-03-22T05:15:39.992 INFO:journalctl@ceph.mon.c.smithi005.stdout: block_restart_interval: 16 2024-03-22T05:15:39.992 INFO:journalctl@ceph.mon.c.smithi005.stdout: index_block_restart_interval: 1 2024-03-22T05:15:39.993 INFO:journalctl@ceph.mon.c.smithi005.stdout: metadata_block_size: 4096 2024-03-22T05:15:39.993 INFO:journalctl@ceph.mon.c.smithi005.stdout: partition_filters: 0 2024-03-22T05:15:39.993 INFO:journalctl@ceph.mon.c.smithi005.stdout: use_delta_encoding: 1 2024-03-22T05:15:39.993 INFO:journalctl@ceph.mon.c.smithi005.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-03-22T05:15:39.993 INFO:journalctl@ceph.mon.c.smithi005.stdout: whole_key_filtering: 1 2024-03-22T05:15:39.993 INFO:journalctl@ceph.mon.c.smithi005.stdout: verify_compression: 0 2024-03-22T05:15:39.993 INFO:journalctl@ceph.mon.c.smithi005.stdout: read_amp_bytes_per_bit: 0 2024-03-22T05:15:39.993 INFO:journalctl@ceph.mon.c.smithi005.stdout: format_version: 4 2024-03-22T05:15:39.993 INFO:journalctl@ceph.mon.c.smithi005.stdout: enable_index_compression: 1 2024-03-22T05:15:39.993 INFO:journalctl@ceph.mon.c.smithi005.stdout: block_align: 0 2024-03-22T05:15:39.993 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.write_buffer_size: 33554432 2024-03-22T05:15:39.993 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.max_write_buffer_number: 2 2024-03-22T05:15:39.994 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.compression: NoCompression 2024-03-22T05:15:39.994 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.bottommost_compression: Disabled 2024-03-22T05:15:39.994 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.prefix_extractor: nullptr 2024-03-22T05:15:39.994 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-22T05:15:39.994 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.num_levels: 7 2024-03-22T05:15:39.994 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-22T05:15:39.994 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-22T05:15:39.994 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-22T05:15:39.994 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-22T05:15:39.994 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-22T05:15:39.994 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-22T05:15:39.995 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-22T05:15:39.995 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-22T05:15:39.995 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-22T05:15:39.995 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-22T05:15:39.995 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-22T05:15:39.995 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.compression_opts.level: 32767 2024-03-22T05:15:39.995 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.compression_opts.strategy: 0 2024-03-22T05:15:39.995 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-22T05:15:39.995 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-22T05:15:39.995 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-22T05:15:39.995 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.compression_opts.enabled: false 2024-03-22T05:15:39.995 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-22T05:15:39.995 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-22T05:15:39.995 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-22T05:15:39.996 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.target_file_size_base: 67108864 2024-03-22T05:15:39.996 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-22T05:15:39.996 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-22T05:15:39.996 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-22T05:15:39.996 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-22T05:15:39.996 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-22T05:15:39.996 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-22T05:15:39.996 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-22T05:15:39.996 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-22T05:15:39.996 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-22T05:15:39.996 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-22T05:15:39.996 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-22T05:15:39.996 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-22T05:15:39.996 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-22T05:15:39.996 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.arena_block_size: 4194304 2024-03-22T05:15:39.996 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-22T05:15:39.997 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-22T05:15:39.997 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-03-22T05:15:39.997 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.disable_auto_compactions: 0 2024-03-22T05:15:39.997 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-22T05:15:39.997 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-22T05:15:39.997 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-22T05:15:39.997 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-22T05:15:39.997 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-22T05:15:39.997 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-22T05:15:39.997 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-22T05:15:39.997 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-22T05:15:39.997 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-22T05:15:39.997 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-22T05:15:39.997 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.table_properties_collectors: 2024-03-22T05:15:39.998 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.inplace_update_support: 0 2024-03-22T05:15:39.998 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-22T05:15:39.998 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-22T05:15:39.998 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-22T05:15:39.998 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-22T05:15:39.998 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.bloom_locality: 0 2024-03-22T05:15:39.998 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.max_successive_merges: 0 2024-03-22T05:15:39.998 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-22T05:15:39.998 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.paranoid_file_checks: 0 2024-03-22T05:15:39.998 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.force_consistency_checks: 1 2024-03-22T05:15:39.998 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.report_bg_io_stats: 0 2024-03-22T05:15:39.998 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.ttl: 2592000 2024-03-22T05:15:39.998 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-22T05:15:39.998 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.enable_blob_files: false 2024-03-22T05:15:39.999 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.min_blob_size: 0 2024-03-22T05:15:39.999 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.blob_file_size: 268435456 2024-03-22T05:15:39.999 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-22T05:15:39.999 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-22T05:15:39.999 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-22T05:15:39.999 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-22T05:15:39.999 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 0 2024-03-22T05:15:39.999 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: [db/version_set.cc:4083] Creating manifest 7 2024-03-22T05:15:39.999 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711084539926287, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-22T05:15:39.999 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #4 mode 2 2024-03-22T05:15:39.999 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-03-22T05:15:39.999 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711084539926598, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1545, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1711084539, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "967946cf-0413-42bf-a64d-b7ecf40eb3bb", "db_session_id": "6I64R0QV82F4NIN8RJTW"}} 2024-03-22T05:15:40.000 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: [db/version_set.cc:4083] Creating manifest 9 2024-03-22T05:15:40.000 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711084539926923, "job": 1, "event": "recovery_finished"} 2024-03-22T05:15:40.000 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-22T05:15:40.000 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x55cb98ad6a80 2024-03-22T05:15:40.000 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: DB pointer 0x55cb98ae6000 2024-03-22T05:15:40.000 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-03-22T05:15:40.000 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-03-22T05:15:40.000 INFO:journalctl@ceph.mon.c.smithi005.stdout: ** DB Stats ** 2024-03-22T05:15:40.000 INFO:journalctl@ceph.mon.c.smithi005.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-22T05:15:40.000 INFO:journalctl@ceph.mon.c.smithi005.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-22T05:15:40.000 INFO:journalctl@ceph.mon.c.smithi005.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-22T05:15:40.000 INFO:journalctl@ceph.mon.c.smithi005.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-22T05:15:40.000 INFO:journalctl@ceph.mon.c.smithi005.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-22T05:15:40.000 INFO:journalctl@ceph.mon.c.smithi005.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-03-22T05:15:40.000 INFO:journalctl@ceph.mon.c.smithi005.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-22T05:15:40.001 INFO:journalctl@ceph.mon.c.smithi005.stdout: 2024-03-22T05:15:40.001 INFO:journalctl@ceph.mon.c.smithi005.stdout: ** Compaction Stats [default] ** 2024-03-22T05:15:40.001 INFO:journalctl@ceph.mon.c.smithi005.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-22T05:15:40.001 INFO:journalctl@ceph.mon.c.smithi005.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-22T05:15:40.001 INFO:journalctl@ceph.mon.c.smithi005.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.0 0.00 0.00 1 0.000 0 0 2024-03-22T05:15:40.001 INFO:journalctl@ceph.mon.c.smithi005.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.0 0.00 0.00 1 0.000 0 0 2024-03-22T05:15:40.001 INFO:journalctl@ceph.mon.c.smithi005.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.0 0.00 0.00 1 0.000 0 0 2024-03-22T05:15:40.001 INFO:journalctl@ceph.mon.c.smithi005.stdout: 2024-03-22T05:15:40.001 INFO:journalctl@ceph.mon.c.smithi005.stdout: ** Compaction Stats [default] ** 2024-03-22T05:15:40.001 INFO:journalctl@ceph.mon.c.smithi005.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-22T05:15:40.001 INFO:journalctl@ceph.mon.c.smithi005.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-22T05:15:40.001 INFO:journalctl@ceph.mon.c.smithi005.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.0 0.00 0.00 1 0.000 0 0 2024-03-22T05:15:40.001 INFO:journalctl@ceph.mon.c.smithi005.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-22T05:15:40.001 INFO:journalctl@ceph.mon.c.smithi005.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-22T05:15:40.002 INFO:journalctl@ceph.mon.c.smithi005.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-22T05:15:40.002 INFO:journalctl@ceph.mon.c.smithi005.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-22T05:15:40.002 INFO:journalctl@ceph.mon.c.smithi005.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-22T05:15:40.002 INFO:journalctl@ceph.mon.c.smithi005.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-22T05:15:40.002 INFO:journalctl@ceph.mon.c.smithi005.stdout: Cumulative compaction: 0.00 GB write, 0.58 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-22T05:15:40.002 INFO:journalctl@ceph.mon.c.smithi005.stdout: Interval compaction: 0.00 GB write, 0.58 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-22T05:15:40.002 INFO:journalctl@ceph.mon.c.smithi005.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-22T05:15:40.003 INFO:journalctl@ceph.mon.c.smithi005.stdout: 2024-03-22T05:15:40.003 INFO:journalctl@ceph.mon.c.smithi005.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-22T05:15:40.003 INFO:journalctl@ceph.mon.c.smithi005.stdout: 2024-03-22T05:15:40.003 INFO:journalctl@ceph.mon.c.smithi005.stdout: ** Compaction Stats [default] ** 2024-03-22T05:15:40.003 INFO:journalctl@ceph.mon.c.smithi005.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-22T05:15:40.003 INFO:journalctl@ceph.mon.c.smithi005.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-22T05:15:40.003 INFO:journalctl@ceph.mon.c.smithi005.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.0 0.00 0.00 1 0.000 0 0 2024-03-22T05:15:40.003 INFO:journalctl@ceph.mon.c.smithi005.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.0 0.00 0.00 1 0.000 0 0 2024-03-22T05:15:40.003 INFO:journalctl@ceph.mon.c.smithi005.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-03-22T05:15:40.003 INFO:journalctl@ceph.mon.c.smithi005.stdout: 2024-03-22T05:15:40.003 INFO:journalctl@ceph.mon.c.smithi005.stdout: ** Compaction Stats [default] ** 2024-03-22T05:15:40.003 INFO:journalctl@ceph.mon.c.smithi005.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-22T05:15:40.003 INFO:journalctl@ceph.mon.c.smithi005.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-22T05:15:40.003 INFO:journalctl@ceph.mon.c.smithi005.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.0 0.00 0.00 1 0.000 0 0 2024-03-22T05:15:40.003 INFO:journalctl@ceph.mon.c.smithi005.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-22T05:15:40.004 INFO:journalctl@ceph.mon.c.smithi005.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-22T05:15:40.004 INFO:journalctl@ceph.mon.c.smithi005.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-22T05:15:40.004 INFO:journalctl@ceph.mon.c.smithi005.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-22T05:15:40.004 INFO:journalctl@ceph.mon.c.smithi005.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-22T05:15:40.004 INFO:journalctl@ceph.mon.c.smithi005.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-22T05:15:40.004 INFO:journalctl@ceph.mon.c.smithi005.stdout: Cumulative compaction: 0.00 GB write, 0.58 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-22T05:15:40.004 INFO:journalctl@ceph.mon.c.smithi005.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-22T05:15:40.004 INFO:journalctl@ceph.mon.c.smithi005.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-22T05:15:40.004 INFO:journalctl@ceph.mon.c.smithi005.stdout: 2024-03-22T05:15:40.004 INFO:journalctl@ceph.mon.c.smithi005.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-22T05:15:40.004 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-03-22T05:15:40.004 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: using public_addrv [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-03-22T05:15:40.004 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: starting mon.c rank -1 at public addrs [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] at bind addrs [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid f59f36e2-e80a-11ee-95cd-87774f69a715 2024-03-22T05:15:40.004 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: mon.c@-1(???) e0 preinit fsid f59f36e2-e80a-11ee-95cd-87774f69a715 2024-03-22T05:15:40.005 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: mon.c@-1(synchronizing).mds e1 new map 2024-03-22T05:15:40.005 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: mon.c@-1(synchronizing).mds e1 print_map 2024-03-22T05:15:40.005 INFO:journalctl@ceph.mon.c.smithi005.stdout: e1 2024-03-22T05:15:40.005 INFO:journalctl@ceph.mon.c.smithi005.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-22T05:15:40.005 INFO:journalctl@ceph.mon.c.smithi005.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T05:15:40.005 INFO:journalctl@ceph.mon.c.smithi005.stdout: legacy client fscid: -1 2024-03-22T05:15:40.005 INFO:journalctl@ceph.mon.c.smithi005.stdout: 2024-03-22T05:15:40.005 INFO:journalctl@ceph.mon.c.smithi005.stdout: No filesystems configured 2024-03-22T05:15:40.005 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-22T05:15:40.005 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-22T05:15:40.005 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-22T05:15:40.005 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-22T05:15:40.005 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-22T05:15:40.005 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-22T05:15:40.005 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-22T05:15:40.006 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-22T05:15:40.006 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-22T05:15:40.006 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-22T05:15:40.006 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T05:15:40.006 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: from='client.? 172.21.15.186:0/1576193235' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-22T05:15:40.006 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-22T05:15:40.006 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 ceph-mon[29523]: mon.c@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete,mon_cluster_log_to_file 2024-03-22T05:15:40.006 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 bash[29504]: 00ec24d38612c9e5b2bd914999a6c666bcc53331ecb8cfd79900c97c98e16c16 2024-03-22T05:15:40.006 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:39 smithi005 systemd[1]: Started Ceph mon.c for f59f36e2-e80a-11ee-95cd-87774f69a715. 2024-03-22T05:15:40.194 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-22T05:15:40.194 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-22T05:15:45.344 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-22T05:15:45.345 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":2,"fsid":"f59f36e2-e80a-11ee-95cd-87774f69a715","modified":"2024-03-22T05:15:39.944561Z","created":"2024-03-22T05:14:16.214419Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3301","nonce":0},{"type":"v1","addr":"172.21.15.5:6790","nonce":0}]},"addr":"172.21.15.5:6790/0","public_addr":"172.21.15.5:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-22T05:15:45.346 INFO:teuthology.orchestra.run.smithi186.stderr:dumped monmap epoch 2 2024-03-22T05:15:45.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:45 smithi005 ceph-mon[29523]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T05:15:45.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:45 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-22T05:15:45.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:45 smithi005 ceph-mon[29523]: mon.a calling monitor election 2024-03-22T05:15:45.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:45 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T05:15:45.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:45 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T05:15:45.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:45 smithi005 ceph-mon[29523]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T05:15:45.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:45 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T05:15:45.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:45 smithi005 ceph-mon[29523]: mon.c calling monitor election 2024-03-22T05:15:45.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:45 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T05:15:45.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:45 smithi005 ceph-mon[29523]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T05:15:45.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:45 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T05:15:45.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:45 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T05:15:45.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:45 smithi005 ceph-mon[29523]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-22T05:15:45.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:45 smithi005 ceph-mon[29523]: monmap e2: 2 mons at {a=[v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0],c=[v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0]} 2024-03-22T05:15:45.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:45 smithi005 ceph-mon[29523]: fsmap 2024-03-22T05:15:45.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:45 smithi005 ceph-mon[29523]: osdmap e4: 0 total, 0 up, 0 in 2024-03-22T05:15:45.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:45 smithi005 ceph-mon[29523]: mgrmap e13: y(active, since 35s) 2024-03-22T05:15:45.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:45 smithi005 ceph-mon[29523]: overall HEALTH_OK 2024-03-22T05:15:45.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:45 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:15:45.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:45 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-22T05:15:45.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:45 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:15:45.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:45 smithi005 ceph-mon[24793]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T05:15:45.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:45 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-22T05:15:45.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:45 smithi005 ceph-mon[24793]: mon.a calling monitor election 2024-03-22T05:15:45.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:45 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T05:15:45.455 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:45 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T05:15:45.455 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:45 smithi005 ceph-mon[24793]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T05:15:45.455 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:45 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T05:15:45.455 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:45 smithi005 ceph-mon[24793]: mon.c calling monitor election 2024-03-22T05:15:45.455 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:45 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T05:15:45.455 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:45 smithi005 ceph-mon[24793]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T05:15:45.455 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:45 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T05:15:45.455 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:45 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T05:15:45.455 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:45 smithi005 ceph-mon[24793]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-22T05:15:45.455 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:45 smithi005 ceph-mon[24793]: monmap e2: 2 mons at {a=[v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0],c=[v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0]} 2024-03-22T05:15:45.455 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:45 smithi005 ceph-mon[24793]: fsmap 2024-03-22T05:15:45.455 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:45 smithi005 ceph-mon[24793]: osdmap e4: 0 total, 0 up, 0 in 2024-03-22T05:15:45.455 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:45 smithi005 ceph-mon[24793]: mgrmap e13: y(active, since 35s) 2024-03-22T05:15:45.455 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:45 smithi005 ceph-mon[24793]: overall HEALTH_OK 2024-03-22T05:15:45.455 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:45 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:15:45.455 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:45 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-22T05:15:45.456 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:45 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:15:46.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:46 smithi005 ceph-mon[24793]: Deploying daemon mon.b on smithi186 2024-03-22T05:15:46.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:46 smithi005 ceph-mon[24793]: from='client.? 172.21.15.186:0/1474346992' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-22T05:15:46.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:46 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T05:15:46.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:46 smithi005 ceph-mon[29523]: Deploying daemon mon.b on smithi186 2024-03-22T05:15:46.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:46 smithi005 ceph-mon[29523]: from='client.? 172.21.15.186:0/1474346992' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-22T05:15:46.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:46 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T05:15:47.401 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-22T05:15:47.402 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-22T05:15:47.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:47 smithi005 ceph-mon[24793]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T05:15:47.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:47 smithi005 ceph-mon[29523]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T05:15:48.768 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:48 smithi186 systemd[1]: Starting Ceph mon.b for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:15:49.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:49 smithi005 ceph-mon[24793]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T05:15:49.480 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:49 smithi005 ceph-mon[29523]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T05:15:49.708 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-22T05:15:49.708 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":2,"fsid":"f59f36e2-e80a-11ee-95cd-87774f69a715","modified":"2024-03-22T05:15:39.944561Z","created":"2024-03-22T05:14:16.214419Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3301","nonce":0},{"type":"v1","addr":"172.21.15.5:6790","nonce":0}]},"addr":"172.21.15.5:6790/0","public_addr":"172.21.15.5:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-22T05:15:49.710 INFO:teuthology.orchestra.run.smithi186.stderr:dumped monmap epoch 2 2024-03-22T05:15:50.017 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:49 smithi186 podman[25685]: 2024-03-22 05:15:49.598370514 +0000 UTC m=+0.025066489 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-03-22T05:15:50.017 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:49 smithi186 podman[25685]: 2024-03-22 05:15:49.72257342 +0000 UTC m=+0.149269388 container create 323f53f932f651b46684b8054199cd7093279274961dcca1ae6995158bfb3583 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mon-b, vcs-type=git, RELEASE=HEAD, release=754, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, architecture=x86_64, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, version=8, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0) 2024-03-22T05:15:50.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:50 smithi005 ceph-mon[24793]: from='client.? 172.21.15.186:0/3287394100' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-22T05:15:50.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:50 smithi005 ceph-mon[29523]: from='client.? 172.21.15.186:0/3287394100' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-22T05:15:51.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 podman[25685]: 2024-03-22 05:15:50.553280351 +0000 UTC m=+0.979976328 container init 323f53f932f651b46684b8054199cd7093279274961dcca1ae6995158bfb3583 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mon-b, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, release=754, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=, vcs-type=git) 2024-03-22T05:15:51.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 podman[25685]: 2024-03-22 05:15:50.556614396 +0000 UTC m=+0.983310362 container start 323f53f932f651b46684b8054199cd7093279274961dcca1ae6995158bfb3583 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mon-b, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, vcs-type=git, io.openshift.expose-services=, name=centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, io.buildah.version=1.19.8, architecture=x86_64, GIT_CLEAN=True) 2024-03-22T05:15:51.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: set uid:gid to 167:167 (ceph:ceph) 2024-03-22T05:15:51.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-03-22T05:15:51.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: pidfile_write: ignore empty --pid-file 2024-03-22T05:15:51.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: load: jerasure load: lrc 2024-03-22T05:15:51.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: RocksDB version: 6.15.5 2024-03-22T05:15:51.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-03-22T05:15:51.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Compile date Apr 18 2022 2024-03-22T05:15:51.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: DB SUMMARY 2024-03-22T05:15:51.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: DB Session ID: TBWBNPGLU8332UF4YATU 2024-03-22T05:15:51.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: CURRENT file: CURRENT 2024-03-22T05:15:51.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: IDENTITY file: IDENTITY 2024-03-22T05:15:51.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2024-03-22T05:15:51.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-03-22T05:15:51.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-03-22T05:15:51.046 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.error_if_exists: 0 2024-03-22T05:15:51.046 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.create_if_missing: 0 2024-03-22T05:15:51.046 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.paranoid_checks: 1 2024-03-22T05:15:51.046 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-22T05:15:51.046 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.env: 0x5602250c6860 2024-03-22T05:15:51.046 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.fs: Posix File System 2024-03-22T05:15:51.046 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.info_log: 0x5602270bbdc0 2024-03-22T05:15:51.046 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.max_file_opening_threads: 16 2024-03-22T05:15:51.046 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.statistics: (nil) 2024-03-22T05:15:51.046 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.use_fsync: 0 2024-03-22T05:15:51.046 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.max_log_file_size: 0 2024-03-22T05:15:51.046 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-22T05:15:51.046 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-22T05:15:51.046 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.keep_log_file_num: 1000 2024-03-22T05:15:51.047 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.recycle_log_file_num: 0 2024-03-22T05:15:51.047 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.allow_fallocate: 1 2024-03-22T05:15:51.047 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.allow_mmap_reads: 0 2024-03-22T05:15:51.047 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.allow_mmap_writes: 0 2024-03-22T05:15:51.047 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.use_direct_reads: 0 2024-03-22T05:15:51.047 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-22T05:15:51.047 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.create_missing_column_families: 0 2024-03-22T05:15:51.047 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.db_log_dir: 2024-03-22T05:15:51.047 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-b/store.db 2024-03-22T05:15:51.047 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-22T05:15:51.047 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-22T05:15:51.047 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-22T05:15:51.047 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-22T05:15:51.047 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-22T05:15:51.047 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-22T05:15:51.047 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.advise_random_on_open: 1 2024-03-22T05:15:51.047 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.db_write_buffer_size: 0 2024-03-22T05:15:51.048 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.write_buffer_manager: 0x5602271ac240 2024-03-22T05:15:51.048 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-22T05:15:51.048 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-03-22T05:15:51.048 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-22T05:15:51.048 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-22T05:15:51.048 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.rate_limiter: (nil) 2024-03-22T05:15:51.048 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-22T05:15:51.048 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.wal_recovery_mode: 2 2024-03-22T05:15:51.048 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.enable_thread_tracking: 0 2024-03-22T05:15:51.048 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.enable_pipelined_write: 0 2024-03-22T05:15:51.048 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.unordered_write: 0 2024-03-22T05:15:51.048 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-22T05:15:51.048 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-22T05:15:51.048 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-22T05:15:51.048 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-22T05:15:51.048 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.row_cache: None 2024-03-22T05:15:51.048 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.wal_filter: None 2024-03-22T05:15:51.049 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-22T05:15:51.049 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.allow_ingest_behind: 0 2024-03-22T05:15:51.049 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.preserve_deletes: 0 2024-03-22T05:15:51.049 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.two_write_queues: 0 2024-03-22T05:15:51.049 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.manual_wal_flush: 0 2024-03-22T05:15:51.049 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.atomic_flush: 0 2024-03-22T05:15:51.049 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-22T05:15:51.049 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-22T05:15:51.049 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-22T05:15:51.049 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.log_readahead_size: 0 2024-03-22T05:15:51.049 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-22T05:15:51.049 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.best_efforts_recovery: 0 2024-03-22T05:15:51.049 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-22T05:15:51.049 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-22T05:15:51.049 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.allow_data_in_errors: 0 2024-03-22T05:15:51.049 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.db_host_id: __hostname__ 2024-03-22T05:15:51.050 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.max_background_jobs: 2 2024-03-22T05:15:51.050 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.max_background_compactions: -1 2024-03-22T05:15:51.050 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.max_subcompactions: 1 2024-03-22T05:15:51.050 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-22T05:15:51.050 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-22T05:15:51.050 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-22T05:15:51.050 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.max_total_wal_size: 0 2024-03-22T05:15:51.050 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-22T05:15:51.050 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-22T05:15:51.050 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-22T05:15:51.050 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-22T05:15:51.050 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.max_open_files: -1 2024-03-22T05:15:51.050 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.bytes_per_sync: 0 2024-03-22T05:15:51.050 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-22T05:15:51.051 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-22T05:15:51.051 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.compaction_readahead_size: 0 2024-03-22T05:15:51.051 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.max_background_flushes: -1 2024-03-22T05:15:51.051 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Compression algorithms supported: 2024-03-22T05:15:51.051 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-22T05:15:51.051 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: kZSTD supported: 0 2024-03-22T05:15:51.051 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: kXpressCompression supported: 0 2024-03-22T05:15:51.051 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: kLZ4HCCompression supported: 1 2024-03-22T05:15:51.051 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: kLZ4Compression supported: 1 2024-03-22T05:15:51.051 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: kBZip2Compression supported: 0 2024-03-22T05:15:51.051 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: kZlibCompression supported: 1 2024-03-22T05:15:51.051 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: kSnappyCompression supported: 1 2024-03-22T05:15:51.051 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-22T05:15:51.051 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000003 2024-03-22T05:15:51.052 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-03-22T05:15:51.052 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-22T05:15:51.052 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.merge_operator: 2024-03-22T05:15:51.052 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.compaction_filter: None 2024-03-22T05:15:51.052 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.compaction_filter_factory: None 2024-03-22T05:15:51.052 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.sst_partitioner_factory: None 2024-03-22T05:15:51.052 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-22T05:15:51.052 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-22T05:15:51.052 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x560227087d00) 2024-03-22T05:15:51.052 INFO:journalctl@ceph.mon.b.smithi186.stdout: cache_index_and_filter_blocks: 1 2024-03-22T05:15:51.052 INFO:journalctl@ceph.mon.b.smithi186.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-22T05:15:51.052 INFO:journalctl@ceph.mon.b.smithi186.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-22T05:15:51.052 INFO:journalctl@ceph.mon.b.smithi186.stdout: pin_top_level_index_and_filter: 1 2024-03-22T05:15:51.052 INFO:journalctl@ceph.mon.b.smithi186.stdout: index_type: 0 2024-03-22T05:15:51.052 INFO:journalctl@ceph.mon.b.smithi186.stdout: data_block_index_type: 0 2024-03-22T05:15:51.052 INFO:journalctl@ceph.mon.b.smithi186.stdout: index_shortening: 1 2024-03-22T05:15:51.052 INFO:journalctl@ceph.mon.b.smithi186.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-22T05:15:51.053 INFO:journalctl@ceph.mon.b.smithi186.stdout: hash_index_allow_collision: 1 2024-03-22T05:15:51.053 INFO:journalctl@ceph.mon.b.smithi186.stdout: checksum: 1 2024-03-22T05:15:51.053 INFO:journalctl@ceph.mon.b.smithi186.stdout: no_block_cache: 0 2024-03-22T05:15:51.053 INFO:journalctl@ceph.mon.b.smithi186.stdout: block_cache: 0x5602270f2170 2024-03-22T05:15:51.053 INFO:journalctl@ceph.mon.b.smithi186.stdout: block_cache_name: BinnedLRUCache 2024-03-22T05:15:51.053 INFO:journalctl@ceph.mon.b.smithi186.stdout: block_cache_options: 2024-03-22T05:15:51.053 INFO:journalctl@ceph.mon.b.smithi186.stdout: capacity : 536870912 2024-03-22T05:15:51.053 INFO:journalctl@ceph.mon.b.smithi186.stdout: num_shard_bits : 4 2024-03-22T05:15:51.053 INFO:journalctl@ceph.mon.b.smithi186.stdout: strict_capacity_limit : 0 2024-03-22T05:15:51.053 INFO:journalctl@ceph.mon.b.smithi186.stdout: high_pri_pool_ratio: 0.000 2024-03-22T05:15:51.053 INFO:journalctl@ceph.mon.b.smithi186.stdout: block_cache_compressed: (nil) 2024-03-22T05:15:51.053 INFO:journalctl@ceph.mon.b.smithi186.stdout: persistent_cache: (nil) 2024-03-22T05:15:51.053 INFO:journalctl@ceph.mon.b.smithi186.stdout: block_size: 4096 2024-03-22T05:15:51.053 INFO:journalctl@ceph.mon.b.smithi186.stdout: block_size_deviation: 10 2024-03-22T05:15:51.053 INFO:journalctl@ceph.mon.b.smithi186.stdout: block_restart_interval: 16 2024-03-22T05:15:51.053 INFO:journalctl@ceph.mon.b.smithi186.stdout: index_block_restart_interval: 1 2024-03-22T05:15:51.053 INFO:journalctl@ceph.mon.b.smithi186.stdout: metadata_block_size: 4096 2024-03-22T05:15:51.054 INFO:journalctl@ceph.mon.b.smithi186.stdout: partition_filters: 0 2024-03-22T05:15:51.054 INFO:journalctl@ceph.mon.b.smithi186.stdout: use_delta_encoding: 1 2024-03-22T05:15:51.054 INFO:journalctl@ceph.mon.b.smithi186.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-03-22T05:15:51.054 INFO:journalctl@ceph.mon.b.smithi186.stdout: whole_key_filtering: 1 2024-03-22T05:15:51.054 INFO:journalctl@ceph.mon.b.smithi186.stdout: verify_compression: 0 2024-03-22T05:15:51.054 INFO:journalctl@ceph.mon.b.smithi186.stdout: read_amp_bytes_per_bit: 0 2024-03-22T05:15:51.054 INFO:journalctl@ceph.mon.b.smithi186.stdout: format_version: 4 2024-03-22T05:15:51.054 INFO:journalctl@ceph.mon.b.smithi186.stdout: enable_index_compression: 1 2024-03-22T05:15:51.054 INFO:journalctl@ceph.mon.b.smithi186.stdout: block_align: 0 2024-03-22T05:15:51.054 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.write_buffer_size: 33554432 2024-03-22T05:15:51.054 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.max_write_buffer_number: 2 2024-03-22T05:15:51.054 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.compression: NoCompression 2024-03-22T05:15:51.054 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.bottommost_compression: Disabled 2024-03-22T05:15:51.054 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.prefix_extractor: nullptr 2024-03-22T05:15:51.055 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-22T05:15:51.055 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.num_levels: 7 2024-03-22T05:15:51.055 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-22T05:15:51.055 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-22T05:15:51.055 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-22T05:15:51.055 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-22T05:15:51.055 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-22T05:15:51.055 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-22T05:15:51.055 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-22T05:15:51.055 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-22T05:15:51.055 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-22T05:15:51.055 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-22T05:15:51.055 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-22T05:15:51.055 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.compression_opts.level: 32767 2024-03-22T05:15:51.055 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.compression_opts.strategy: 0 2024-03-22T05:15:51.055 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-22T05:15:51.056 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-22T05:15:51.056 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-22T05:15:51.056 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.compression_opts.enabled: false 2024-03-22T05:15:51.057 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-22T05:15:51.057 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-22T05:15:51.057 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-22T05:15:51.057 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.target_file_size_base: 67108864 2024-03-22T05:15:51.057 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-22T05:15:51.057 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-22T05:15:51.057 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-22T05:15:51.057 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-22T05:15:51.057 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-22T05:15:51.057 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-22T05:15:51.057 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-22T05:15:51.057 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-22T05:15:51.057 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-22T05:15:51.058 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-22T05:15:51.058 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-22T05:15:51.058 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-22T05:15:51.058 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-22T05:15:51.058 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.arena_block_size: 4194304 2024-03-22T05:15:51.058 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-22T05:15:51.058 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-22T05:15:51.058 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-03-22T05:15:51.058 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.disable_auto_compactions: 0 2024-03-22T05:15:51.058 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-22T05:15:51.058 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-22T05:15:51.058 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-22T05:15:51.058 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-22T05:15:51.059 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-22T05:15:51.059 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-22T05:15:51.059 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-22T05:15:51.059 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-22T05:15:51.059 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-22T05:15:51.059 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-22T05:15:51.059 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.table_properties_collectors: 2024-03-22T05:15:51.059 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.inplace_update_support: 0 2024-03-22T05:15:51.059 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-22T05:15:51.059 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-22T05:15:51.059 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-22T05:15:51.059 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-22T05:15:51.059 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.bloom_locality: 0 2024-03-22T05:15:51.059 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.max_successive_merges: 0 2024-03-22T05:15:51.059 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-22T05:15:51.059 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.paranoid_file_checks: 0 2024-03-22T05:15:51.060 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.force_consistency_checks: 1 2024-03-22T05:15:51.060 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.report_bg_io_stats: 0 2024-03-22T05:15:51.060 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.ttl: 2592000 2024-03-22T05:15:51.060 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-22T05:15:51.060 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.enable_blob_files: false 2024-03-22T05:15:51.060 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.min_blob_size: 0 2024-03-22T05:15:51.060 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.blob_file_size: 268435456 2024-03-22T05:15:51.060 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-22T05:15:51.060 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-22T05:15:51.060 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-22T05:15:51.060 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-22T05:15:51.060 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 0 2024-03-22T05:15:51.060 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: [db/version_set.cc:4083] Creating manifest 7 2024-03-22T05:15:51.060 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711084550584137, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-22T05:15:51.060 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #4 mode 2 2024-03-22T05:15:51.060 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-03-22T05:15:51.061 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711084550584446, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1670, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1711084550, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "9e536753-d7a3-4f36-9e19-d49e4b081359", "db_session_id": "TBWBNPGLU8332UF4YATU"}} 2024-03-22T05:15:51.061 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: [db/version_set.cc:4083] Creating manifest 9 2024-03-22T05:15:51.061 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711084550584749, "job": 1, "event": "recovery_finished"} 2024-03-22T05:15:51.061 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-22T05:15:51.061 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x5602270d8a80 2024-03-22T05:15:51.061 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: DB pointer 0x5602270e8000 2024-03-22T05:15:51.061 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-03-22T05:15:51.061 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-03-22T05:15:51.061 INFO:journalctl@ceph.mon.b.smithi186.stdout: ** DB Stats ** 2024-03-22T05:15:51.061 INFO:journalctl@ceph.mon.b.smithi186.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-22T05:15:51.061 INFO:journalctl@ceph.mon.b.smithi186.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-22T05:15:51.061 INFO:journalctl@ceph.mon.b.smithi186.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-22T05:15:51.061 INFO:journalctl@ceph.mon.b.smithi186.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-22T05:15:51.061 INFO:journalctl@ceph.mon.b.smithi186.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-22T05:15:51.061 INFO:journalctl@ceph.mon.b.smithi186.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-03-22T05:15:51.062 INFO:journalctl@ceph.mon.b.smithi186.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-22T05:15:51.062 INFO:journalctl@ceph.mon.b.smithi186.stdout: 2024-03-22T05:15:51.062 INFO:journalctl@ceph.mon.b.smithi186.stdout: ** Compaction Stats [default] ** 2024-03-22T05:15:51.062 INFO:journalctl@ceph.mon.b.smithi186.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-22T05:15:51.062 INFO:journalctl@ceph.mon.b.smithi186.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-22T05:15:51.062 INFO:journalctl@ceph.mon.b.smithi186.stdout: L0 1/0 1.63 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.4 0.00 0.00 1 0.000 0 0 2024-03-22T05:15:51.062 INFO:journalctl@ceph.mon.b.smithi186.stdout: Sum 1/0 1.63 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.4 0.00 0.00 1 0.000 0 0 2024-03-22T05:15:51.062 INFO:journalctl@ceph.mon.b.smithi186.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.4 0.00 0.00 1 0.000 0 0 2024-03-22T05:15:51.062 INFO:journalctl@ceph.mon.b.smithi186.stdout: 2024-03-22T05:15:51.062 INFO:journalctl@ceph.mon.b.smithi186.stdout: ** Compaction Stats [default] ** 2024-03-22T05:15:51.062 INFO:journalctl@ceph.mon.b.smithi186.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-22T05:15:51.062 INFO:journalctl@ceph.mon.b.smithi186.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-22T05:15:51.062 INFO:journalctl@ceph.mon.b.smithi186.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.4 0.00 0.00 1 0.000 0 0 2024-03-22T05:15:51.063 INFO:journalctl@ceph.mon.b.smithi186.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-22T05:15:51.063 INFO:journalctl@ceph.mon.b.smithi186.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-22T05:15:51.063 INFO:journalctl@ceph.mon.b.smithi186.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-22T05:15:51.063 INFO:journalctl@ceph.mon.b.smithi186.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-22T05:15:51.063 INFO:journalctl@ceph.mon.b.smithi186.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-22T05:15:51.063 INFO:journalctl@ceph.mon.b.smithi186.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-22T05:15:51.063 INFO:journalctl@ceph.mon.b.smithi186.stdout: Cumulative compaction: 0.00 GB write, 0.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-22T05:15:51.063 INFO:journalctl@ceph.mon.b.smithi186.stdout: Interval compaction: 0.00 GB write, 0.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-22T05:15:51.063 INFO:journalctl@ceph.mon.b.smithi186.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-22T05:15:51.063 INFO:journalctl@ceph.mon.b.smithi186.stdout: 2024-03-22T05:15:51.063 INFO:journalctl@ceph.mon.b.smithi186.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-22T05:15:51.063 INFO:journalctl@ceph.mon.b.smithi186.stdout: 2024-03-22T05:15:51.063 INFO:journalctl@ceph.mon.b.smithi186.stdout: ** Compaction Stats [default] ** 2024-03-22T05:15:51.063 INFO:journalctl@ceph.mon.b.smithi186.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-22T05:15:51.063 INFO:journalctl@ceph.mon.b.smithi186.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-22T05:15:51.064 INFO:journalctl@ceph.mon.b.smithi186.stdout: L0 1/0 1.63 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.4 0.00 0.00 1 0.000 0 0 2024-03-22T05:15:51.064 INFO:journalctl@ceph.mon.b.smithi186.stdout: Sum 1/0 1.63 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.4 0.00 0.00 1 0.000 0 0 2024-03-22T05:15:51.064 INFO:journalctl@ceph.mon.b.smithi186.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-03-22T05:15:51.064 INFO:journalctl@ceph.mon.b.smithi186.stdout: 2024-03-22T05:15:51.064 INFO:journalctl@ceph.mon.b.smithi186.stdout: ** Compaction Stats [default] ** 2024-03-22T05:15:51.064 INFO:journalctl@ceph.mon.b.smithi186.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-22T05:15:51.064 INFO:journalctl@ceph.mon.b.smithi186.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-22T05:15:51.064 INFO:journalctl@ceph.mon.b.smithi186.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.4 0.00 0.00 1 0.000 0 0 2024-03-22T05:15:51.064 INFO:journalctl@ceph.mon.b.smithi186.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-22T05:15:51.064 INFO:journalctl@ceph.mon.b.smithi186.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-22T05:15:51.064 INFO:journalctl@ceph.mon.b.smithi186.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-22T05:15:51.064 INFO:journalctl@ceph.mon.b.smithi186.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-22T05:15:51.064 INFO:journalctl@ceph.mon.b.smithi186.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-22T05:15:51.064 INFO:journalctl@ceph.mon.b.smithi186.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-22T05:15:51.064 INFO:journalctl@ceph.mon.b.smithi186.stdout: Cumulative compaction: 0.00 GB write, 0.63 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-22T05:15:51.064 INFO:journalctl@ceph.mon.b.smithi186.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-22T05:15:51.065 INFO:journalctl@ceph.mon.b.smithi186.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-22T05:15:51.065 INFO:journalctl@ceph.mon.b.smithi186.stdout: 2024-03-22T05:15:51.065 INFO:journalctl@ceph.mon.b.smithi186.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-22T05:15:51.065 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-03-22T05:15:51.065 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: using public_addr v2:172.21.15.186:0/0 -> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-03-22T05:15:51.065 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: starting mon.b rank -1 at public addrs [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] at bind addrs [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid f59f36e2-e80a-11ee-95cd-87774f69a715 2024-03-22T05:15:51.065 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: mon.b@-1(???) e0 preinit fsid f59f36e2-e80a-11ee-95cd-87774f69a715 2024-03-22T05:15:51.065 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: mon.b@-1(synchronizing).mds e1 new map 2024-03-22T05:15:51.065 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: mon.b@-1(synchronizing).mds e1 print_map 2024-03-22T05:15:51.065 INFO:journalctl@ceph.mon.b.smithi186.stdout: e1 2024-03-22T05:15:51.065 INFO:journalctl@ceph.mon.b.smithi186.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-22T05:15:51.065 INFO:journalctl@ceph.mon.b.smithi186.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T05:15:51.065 INFO:journalctl@ceph.mon.b.smithi186.stdout: legacy client fscid: -1 2024-03-22T05:15:51.065 INFO:journalctl@ceph.mon.b.smithi186.stdout: 2024-03-22T05:15:51.065 INFO:journalctl@ceph.mon.b.smithi186.stdout: No filesystems configured 2024-03-22T05:15:51.066 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-22T05:15:51.066 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-22T05:15:51.066 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-22T05:15:51.066 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-22T05:15:51.066 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-22T05:15:51.066 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-22T05:15:51.066 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-22T05:15:51.066 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-22T05:15:51.066 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-22T05:15:51.066 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-22T05:15:51.066 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T05:15:51.066 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: from='client.? 172.21.15.186:0/1576193235' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-22T05:15:51.066 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T05:15:51.066 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-22T05:15:51.067 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: mon.a calling monitor election 2024-03-22T05:15:51.067 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T05:15:51.067 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T05:15:51.067 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T05:15:51.067 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T05:15:51.067 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: mon.c calling monitor election 2024-03-22T05:15:51.067 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T05:15:51.067 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T05:15:51.067 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T05:15:51.067 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T05:15:51.067 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-22T05:15:51.067 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: monmap e2: 2 mons at {a=[v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0],c=[v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0]} 2024-03-22T05:15:51.067 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: fsmap 2024-03-22T05:15:51.067 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: osdmap e4: 0 total, 0 up, 0 in 2024-03-22T05:15:51.067 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: mgrmap e13: y(active, since 35s) 2024-03-22T05:15:51.067 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: overall HEALTH_OK 2024-03-22T05:15:51.067 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:15:51.068 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-22T05:15:51.068 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:15:51.068 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: Deploying daemon mon.b on smithi186 2024-03-22T05:15:51.068 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: from='client.? 172.21.15.186:0/1474346992' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-22T05:15:51.068 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T05:15:51.068 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T05:15:51.068 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T05:15:51.068 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: from='client.? 172.21.15.186:0/3287394100' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-22T05:15:51.068 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: mon.b@-1(synchronizing).paxosservice(auth 1..4) refresh upgraded, format 0 -> 3 2024-03-22T05:15:51.068 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 ceph-mon[25720]: mon.b@-1(synchronizing) e2 handle_conf_change mon_allow_pool_delete,mon_cluster_log_to_file 2024-03-22T05:15:51.068 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 bash[25685]: 323f53f932f651b46684b8054199cd7093279274961dcca1ae6995158bfb3583 2024-03-22T05:15:51.068 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:50 smithi186 systemd[1]: Started Ceph mon.b for f59f36e2-e80a-11ee-95cd-87774f69a715. 2024-03-22T05:15:51.316 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-22T05:15:51.316 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-22T05:15:55.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:55 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-22T05:15:55.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:55 smithi005 ceph-mon[24793]: mon.a calling monitor election 2024-03-22T05:15:55.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:55 smithi005 ceph-mon[24793]: mon.c calling monitor election 2024-03-22T05:15:55.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:55 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T05:15:55.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:55 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T05:15:55.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:55 smithi005 ceph-mon[24793]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T05:15:55.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:55 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T05:15:55.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:55 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T05:15:55.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:55 smithi005 ceph-mon[24793]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T05:15:55.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:55 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T05:15:55.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:55 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T05:15:55.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:55 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T05:15:55.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:55 smithi005 ceph-mon[24793]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-22T05:15:55.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:55 smithi005 ceph-mon[24793]: monmap e3: 3 mons at {a=[v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0],b=[v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0],c=[v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0]} 2024-03-22T05:15:55.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:55 smithi005 ceph-mon[24793]: fsmap 2024-03-22T05:15:55.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:55 smithi005 ceph-mon[24793]: osdmap e4: 0 total, 0 up, 0 in 2024-03-22T05:15:55.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:55 smithi005 ceph-mon[24793]: mgrmap e13: y(active, since 46s) 2024-03-22T05:15:55.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:55 smithi005 ceph-mon[24793]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-22T05:15:55.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:55 smithi005 ceph-mon[24793]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-22T05:15:55.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:55 smithi005 ceph-mon[24793]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-22T05:15:55.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:55 smithi005 ceph-mon[24793]: mon.b (rank 2) addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] is down (out of quorum) 2024-03-22T05:15:55.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:55 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:15:55.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:55 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:15:55.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:55 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:15:55.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:55 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:15:55.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:55 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-22T05:15:55.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:55 smithi005 ceph-mon[29523]: mon.a calling monitor election 2024-03-22T05:15:55.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:55 smithi005 ceph-mon[29523]: mon.c calling monitor election 2024-03-22T05:15:55.955 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:55 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T05:15:55.955 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:55 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T05:15:55.955 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:55 smithi005 ceph-mon[29523]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T05:15:55.955 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:55 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T05:15:55.955 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:55 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T05:15:55.955 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:55 smithi005 ceph-mon[29523]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T05:15:55.955 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:55 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T05:15:55.955 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:55 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T05:15:55.955 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:55 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T05:15:55.955 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:55 smithi005 ceph-mon[29523]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-22T05:15:55.955 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:55 smithi005 ceph-mon[29523]: monmap e3: 3 mons at {a=[v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0],b=[v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0],c=[v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0]} 2024-03-22T05:15:55.955 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:55 smithi005 ceph-mon[29523]: fsmap 2024-03-22T05:15:55.955 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:55 smithi005 ceph-mon[29523]: osdmap e4: 0 total, 0 up, 0 in 2024-03-22T05:15:55.955 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:55 smithi005 ceph-mon[29523]: mgrmap e13: y(active, since 46s) 2024-03-22T05:15:55.955 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:55 smithi005 ceph-mon[29523]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-22T05:15:55.956 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:55 smithi005 ceph-mon[29523]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-22T05:15:55.956 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:55 smithi005 ceph-mon[29523]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-22T05:15:55.956 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:55 smithi005 ceph-mon[29523]: mon.b (rank 2) addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] is down (out of quorum) 2024-03-22T05:15:55.956 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:55 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:15:55.956 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:55 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:15:55.956 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:55 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:15:55.956 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:55 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:15:56.153 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-03-22T05:15:56.153 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":3,"fsid":"f59f36e2-e80a-11ee-95cd-87774f69a715","modified":"2024-03-22T05:15:50.603692Z","created":"2024-03-22T05:14:16.214419Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3300","nonce":0},{"type":"v1","addr":"172.21.15.5:6789","nonce":0}]},"addr":"172.21.15.5:6789/0","public_addr":"172.21.15.5:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:3301","nonce":0},{"type":"v1","addr":"172.21.15.5:6790","nonce":0}]},"addr":"172.21.15.5:6790/0","public_addr":"172.21.15.5:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:3300","nonce":0},{"type":"v1","addr":"172.21.15.186:6789","nonce":0}]},"addr":"172.21.15.186:6789/0","public_addr":"172.21.15.186:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-22T05:15:56.154 INFO:teuthology.orchestra.run.smithi186.stderr:dumped monmap epoch 3 2024-03-22T05:15:56.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:56 smithi005 ceph-mon[24793]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T05:15:56.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:56 smithi005 ceph-mon[24793]: from='client.? 172.21.15.186:0/3094000269' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-22T05:15:56.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:56 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T05:15:56.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:56 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:15:56.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:56 smithi005 ceph-mon[29523]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T05:15:56.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:56 smithi005 ceph-mon[29523]: from='client.? 172.21.15.186:0/3094000269' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-22T05:15:56.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:56 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T05:15:56.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:56 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:15:57.152 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-22T05:15:57.152 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -- ceph config generate-minimal-conf 2024-03-22T05:15:57.993 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:57 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-22T05:15:57.993 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:57 smithi186 ceph-mon[25720]: mon.a calling monitor election 2024-03-22T05:15:57.993 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:57 smithi186 ceph-mon[25720]: mon.c calling monitor election 2024-03-22T05:15:57.993 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:57 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T05:15:57.993 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:57 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T05:15:57.994 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:57 smithi186 ceph-mon[25720]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T05:15:57.994 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:57 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T05:15:57.994 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:57 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T05:15:57.994 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:57 smithi186 ceph-mon[25720]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T05:15:57.994 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:57 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T05:15:57.994 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:57 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T05:15:57.994 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:57 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T05:15:57.994 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:57 smithi186 ceph-mon[25720]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-22T05:15:57.994 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:57 smithi186 ceph-mon[25720]: monmap e3: 3 mons at {a=[v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0],b=[v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0],c=[v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0]} 2024-03-22T05:15:57.994 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:57 smithi186 ceph-mon[25720]: fsmap 2024-03-22T05:15:57.994 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:57 smithi186 ceph-mon[25720]: osdmap e4: 0 total, 0 up, 0 in 2024-03-22T05:15:57.994 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:57 smithi186 ceph-mon[25720]: mgrmap e13: y(active, since 46s) 2024-03-22T05:15:57.994 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:57 smithi186 ceph-mon[25720]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-22T05:15:57.995 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:57 smithi186 ceph-mon[25720]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-22T05:15:57.995 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:57 smithi186 ceph-mon[25720]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-22T05:15:57.995 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:57 smithi186 ceph-mon[25720]: mon.b (rank 2) addr [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] is down (out of quorum) 2024-03-22T05:15:57.995 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:57 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:15:57.995 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:57 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:15:57.995 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:57 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:15:57.995 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:57 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:15:57.995 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:57 smithi186 ceph-mon[25720]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T05:15:57.995 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:57 smithi186 ceph-mon[25720]: from='client.? 172.21.15.186:0/3094000269' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-22T05:15:57.995 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:57 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T05:15:57.995 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:57 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:15:57.995 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:57 smithi186 ceph-mon[25720]: mon.b calling monitor election 2024-03-22T05:15:57.995 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:57 smithi186 ceph-mon[25720]: mon.b calling monitor election 2024-03-22T05:15:57.995 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:57 smithi186 ceph-mon[25720]: mon.c calling monitor election 2024-03-22T05:15:57.995 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:57 smithi186 ceph-mon[25720]: mon.a calling monitor election 2024-03-22T05:15:57.995 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:57 smithi186 ceph-mon[25720]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-22T05:15:57.996 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:57 smithi186 ceph-mon[25720]: monmap e3: 3 mons at {a=[v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0],b=[v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0],c=[v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0]} 2024-03-22T05:15:57.996 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:57 smithi186 ceph-mon[25720]: fsmap 2024-03-22T05:15:57.996 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:57 smithi186 ceph-mon[25720]: osdmap e4: 0 total, 0 up, 0 in 2024-03-22T05:15:57.996 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:57 smithi186 ceph-mon[25720]: mgrmap e13: y(active, since 48s) 2024-03-22T05:15:57.996 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:57 smithi186 ceph-mon[25720]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-22T05:15:57.996 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:57 smithi186 ceph-mon[25720]: Cluster is now healthy 2024-03-22T05:15:57.996 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:57 smithi186 ceph-mon[25720]: overall HEALTH_OK 2024-03-22T05:15:58.255 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:57 smithi005 ceph-mon[24793]: mon.b calling monitor election 2024-03-22T05:15:58.255 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:57 smithi005 ceph-mon[24793]: mon.b calling monitor election 2024-03-22T05:15:58.255 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:57 smithi005 ceph-mon[24793]: mon.c calling monitor election 2024-03-22T05:15:58.255 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:57 smithi005 ceph-mon[24793]: mon.a calling monitor election 2024-03-22T05:15:58.255 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:57 smithi005 ceph-mon[24793]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-22T05:15:58.255 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:57 smithi005 ceph-mon[24793]: monmap e3: 3 mons at {a=[v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0],b=[v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0],c=[v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0]} 2024-03-22T05:15:58.255 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:57 smithi005 ceph-mon[24793]: fsmap 2024-03-22T05:15:58.255 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:57 smithi005 ceph-mon[24793]: osdmap e4: 0 total, 0 up, 0 in 2024-03-22T05:15:58.256 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:57 smithi005 ceph-mon[24793]: mgrmap e13: y(active, since 48s) 2024-03-22T05:15:58.256 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:57 smithi005 ceph-mon[24793]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-22T05:15:58.256 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:57 smithi005 ceph-mon[24793]: Cluster is now healthy 2024-03-22T05:15:58.256 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:57 smithi005 ceph-mon[24793]: overall HEALTH_OK 2024-03-22T05:15:58.256 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:57 smithi005 ceph-mon[29523]: mon.b calling monitor election 2024-03-22T05:15:58.256 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:57 smithi005 ceph-mon[29523]: mon.b calling monitor election 2024-03-22T05:15:58.256 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:57 smithi005 ceph-mon[29523]: mon.c calling monitor election 2024-03-22T05:15:58.256 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:57 smithi005 ceph-mon[29523]: mon.a calling monitor election 2024-03-22T05:15:58.256 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:57 smithi005 ceph-mon[29523]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-22T05:15:58.256 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:57 smithi005 ceph-mon[29523]: monmap e3: 3 mons at {a=[v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0],b=[v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0],c=[v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0]} 2024-03-22T05:15:58.256 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:57 smithi005 ceph-mon[29523]: fsmap 2024-03-22T05:15:58.256 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:57 smithi005 ceph-mon[29523]: osdmap e4: 0 total, 0 up, 0 in 2024-03-22T05:15:58.257 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:57 smithi005 ceph-mon[29523]: mgrmap e13: y(active, since 48s) 2024-03-22T05:15:58.257 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:57 smithi005 ceph-mon[29523]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-22T05:15:58.257 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:57 smithi005 ceph-mon[29523]: Cluster is now healthy 2024-03-22T05:15:58.257 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:57 smithi005 ceph-mon[29523]: overall HEALTH_OK 2024-03-22T05:15:59.335 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:59 smithi005 ceph-mon[24793]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T05:15:59.335 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:59 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:15:59.335 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:59 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-22T05:15:59.335 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:59 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-22T05:15:59.335 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:59 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:15:59.335 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:15:59 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T05:15:59.336 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:59 smithi005 ceph-mon[29523]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T05:15:59.336 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:59 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:15:59.336 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:59 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-22T05:15:59.336 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:59 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-22T05:15:59.336 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:59 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:15:59.336 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:15:59 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T05:15:59.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:59 smithi186 ceph-mon[25720]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T05:15:59.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:59 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:15:59.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:59 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-22T05:15:59.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:59 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-22T05:15:59.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:59 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:15:59.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:15:59 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T05:15:59.773 INFO:teuthology.orchestra.run.smithi005.stdout:# minimal ceph.conf for f59f36e2-e80a-11ee-95cd-87774f69a715 2024-03-22T05:15:59.774 INFO:teuthology.orchestra.run.smithi005.stdout:[global] 2024-03-22T05:15:59.774 INFO:teuthology.orchestra.run.smithi005.stdout: fsid = f59f36e2-e80a-11ee-95cd-87774f69a715 2024-03-22T05:15:59.774 INFO:teuthology.orchestra.run.smithi005.stdout: mon_host = [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] 2024-03-22T05:16:00.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:00 smithi005 ceph-mon[24793]: Reconfiguring mon.c (monmap changed)... 2024-03-22T05:16:00.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:00 smithi005 ceph-mon[24793]: Reconfiguring daemon mon.c on smithi005 2024-03-22T05:16:00.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:00 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:16:00.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:00 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/123459246' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:16:00.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:00 smithi005 ceph-mon[29523]: Reconfiguring mon.c (monmap changed)... 2024-03-22T05:16:00.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:00 smithi005 ceph-mon[29523]: Reconfiguring daemon mon.c on smithi005 2024-03-22T05:16:00.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:00 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:16:00.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:00 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/123459246' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:16:00.373 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:00 smithi186 ceph-mon[25720]: Reconfiguring mon.c (monmap changed)... 2024-03-22T05:16:00.373 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:00 smithi186 ceph-mon[25720]: Reconfiguring daemon mon.c on smithi005 2024-03-22T05:16:00.373 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:00 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:16:00.373 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:00 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/123459246' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:16:01.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:01 smithi005 ceph-mon[24793]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T05:16:01.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:01 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:16:01.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:01 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-22T05:16:01.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:01 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-22T05:16:01.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:01 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:16:01.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:01 smithi005 ceph-mon[29523]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T05:16:01.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:01 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:16:01.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:01 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-22T05:16:01.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:01 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-22T05:16:01.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:01 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:16:01.256 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:01 smithi186 ceph-mon[25720]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T05:16:01.257 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:01 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:16:01.257 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:01 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-22T05:16:01.257 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:01 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-22T05:16:01.257 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:01 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:16:01.275 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-22T05:16:01.275 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-03-22T05:16:01.275 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/etc/ceph/ceph.conf 2024-03-22T05:16:01.303 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-03-22T05:16:01.303 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-22T05:16:01.370 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-03-22T05:16:01.370 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/ceph/ceph.conf 2024-03-22T05:16:01.397 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-03-22T05:16:01.398 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-22T05:16:01.464 INFO:tasks.cephadm:Adding mgr.y on smithi005 2024-03-22T05:16:01.464 INFO:tasks.cephadm:Adding mgr.x on smithi186 2024-03-22T05:16:01.464 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -- ceph orch apply mgr '2;smithi005=y;smithi186=x' 2024-03-22T05:16:02.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:02 smithi005 ceph-mon[24793]: Reconfiguring mon.b (monmap changed)... 2024-03-22T05:16:02.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:02 smithi005 ceph-mon[24793]: Reconfiguring daemon mon.b on smithi186 2024-03-22T05:16:02.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:02 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:16:02.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:02 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:16:02.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:02 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:16:02.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:02 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:16:02.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:02 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:16:02.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:02 smithi005 ceph-mon[29523]: Reconfiguring mon.b (monmap changed)... 2024-03-22T05:16:02.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:02 smithi005 ceph-mon[29523]: Reconfiguring daemon mon.b on smithi186 2024-03-22T05:16:02.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:02 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:16:02.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:02 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:16:02.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:02 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:16:02.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:02 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:16:02.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:02 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:16:02.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:02 smithi186 ceph-mon[25720]: Reconfiguring mon.b (monmap changed)... 2024-03-22T05:16:02.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:02 smithi186 ceph-mon[25720]: Reconfiguring daemon mon.b on smithi186 2024-03-22T05:16:02.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:02 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:16:02.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:02 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:16:02.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:02 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:16:02.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:02 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:16:02.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:02 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:16:03.149 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:03 smithi186 ceph-mon[25720]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T05:16:03.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:03 smithi005 ceph-mon[24793]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T05:16:03.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:03 smithi005 ceph-mon[29523]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T05:16:03.541 INFO:teuthology.orchestra.run.smithi186.stdout:Scheduled mgr update... 2024-03-22T05:16:04.153 DEBUG:teuthology.orchestra.run.smithi186:mgr.x> sudo journalctl -f -n 0 -u ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@mgr.x.service 2024-03-22T05:16:04.155 INFO:tasks.cephadm:Deploying OSDs... 2024-03-22T05:16:04.155 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-03-22T05:16:04.156 DEBUG:teuthology.orchestra.run.smithi005:> dd if=/scratch_devs of=/dev/stdout 2024-03-22T05:16:04.174 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-22T05:16:04.175 DEBUG:teuthology.orchestra.run.smithi005:> stat /dev/vg_nvme/lv_1 2024-03-22T05:16:04.232 INFO:teuthology.orchestra.run.smithi005.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-22T05:16:04.232 INFO:teuthology.orchestra.run.smithi005.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-22T05:16:04.232 INFO:teuthology.orchestra.run.smithi005.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-22T05:16:04.232 INFO:teuthology.orchestra.run.smithi005.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-22T05:16:04.232 INFO:teuthology.orchestra.run.smithi005.stdout:Context: system_u:object_r:device_t:s0 2024-03-22T05:16:04.232 INFO:teuthology.orchestra.run.smithi005.stdout:Access: 2024-03-22 05:15:39.454977942 +0000 2024-03-22T05:16:04.232 INFO:teuthology.orchestra.run.smithi005.stdout:Modify: 2024-03-22 05:13:20.533949420 +0000 2024-03-22T05:16:04.232 INFO:teuthology.orchestra.run.smithi005.stdout:Change: 2024-03-22 05:13:20.533949420 +0000 2024-03-22T05:16:04.232 INFO:teuthology.orchestra.run.smithi005.stdout: Birth: 2024-03-22 05:13:20.533949420 +0000 2024-03-22T05:16:04.233 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-22T05:16:04.302 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records in 2024-03-22T05:16:04.302 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records out 2024-03-22T05:16:04.302 INFO:teuthology.orchestra.run.smithi005.stderr:512 bytes copied, 0.000122258 s, 4.2 MB/s 2024-03-22T05:16:04.303 DEBUG:teuthology.orchestra.run.smithi005:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-22T05:16:04.361 DEBUG:teuthology.orchestra.run.smithi005:> stat /dev/vg_nvme/lv_2 2024-03-22T05:16:04.418 INFO:teuthology.orchestra.run.smithi005.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-22T05:16:04.418 INFO:teuthology.orchestra.run.smithi005.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-22T05:16:04.418 INFO:teuthology.orchestra.run.smithi005.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-22T05:16:04.418 INFO:teuthology.orchestra.run.smithi005.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-22T05:16:04.418 INFO:teuthology.orchestra.run.smithi005.stdout:Context: system_u:object_r:device_t:s0 2024-03-22T05:16:04.418 INFO:teuthology.orchestra.run.smithi005.stdout:Access: 2024-03-22 05:15:39.454977942 +0000 2024-03-22T05:16:04.418 INFO:teuthology.orchestra.run.smithi005.stdout:Modify: 2024-03-22 05:13:20.879942381 +0000 2024-03-22T05:16:04.419 INFO:teuthology.orchestra.run.smithi005.stdout:Change: 2024-03-22 05:13:20.879942381 +0000 2024-03-22T05:16:04.419 INFO:teuthology.orchestra.run.smithi005.stdout: Birth: 2024-03-22 05:13:20.879942381 +0000 2024-03-22T05:16:04.419 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-22T05:16:04.488 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records in 2024-03-22T05:16:04.488 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records out 2024-03-22T05:16:04.488 INFO:teuthology.orchestra.run.smithi005.stderr:512 bytes copied, 0.000120901 s, 4.2 MB/s 2024-03-22T05:16:04.489 DEBUG:teuthology.orchestra.run.smithi005:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-22T05:16:04.547 DEBUG:teuthology.orchestra.run.smithi005:> stat /dev/vg_nvme/lv_3 2024-03-22T05:16:04.604 INFO:teuthology.orchestra.run.smithi005.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-22T05:16:04.605 INFO:teuthology.orchestra.run.smithi005.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-22T05:16:04.605 INFO:teuthology.orchestra.run.smithi005.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-22T05:16:04.605 INFO:teuthology.orchestra.run.smithi005.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-22T05:16:04.605 INFO:teuthology.orchestra.run.smithi005.stdout:Context: system_u:object_r:device_t:s0 2024-03-22T05:16:04.605 INFO:teuthology.orchestra.run.smithi005.stdout:Access: 2024-03-22 05:15:39.455977922 +0000 2024-03-22T05:16:04.605 INFO:teuthology.orchestra.run.smithi005.stdout:Modify: 2024-03-22 05:13:21.223935382 +0000 2024-03-22T05:16:04.605 INFO:teuthology.orchestra.run.smithi005.stdout:Change: 2024-03-22 05:13:21.223935382 +0000 2024-03-22T05:16:04.605 INFO:teuthology.orchestra.run.smithi005.stdout: Birth: 2024-03-22 05:13:21.223935382 +0000 2024-03-22T05:16:04.605 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-22T05:16:04.667 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:04 smithi186 ceph-mon[25720]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T05:16:04.667 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:04 smithi186 ceph-mon[25720]: from='client.34103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi005=y;smithi186=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:16:04.667 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:04 smithi186 ceph-mon[25720]: Saving service mgr spec with placement smithi005=y;smithi186=x;count:2 2024-03-22T05:16:04.667 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:04 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:16:04.667 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:04 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:16:04.668 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:04 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:16:04.668 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:04 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:16:04.668 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:04 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:16:04.668 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:04 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-22T05:16:04.668 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:04 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-22T05:16:04.668 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:04 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-22T05:16:04.668 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:04 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:16:04.668 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:04 smithi186 ceph-mon[25720]: Deploying daemon mgr.x on smithi186 2024-03-22T05:16:04.671 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:04 smithi005 ceph-mon[24793]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T05:16:04.671 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:04 smithi005 ceph-mon[24793]: from='client.34103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi005=y;smithi186=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:16:04.671 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:04 smithi005 ceph-mon[24793]: Saving service mgr spec with placement smithi005=y;smithi186=x;count:2 2024-03-22T05:16:04.671 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:04 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:16:04.671 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:04 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:16:04.671 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:04 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:16:04.671 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:04 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:16:04.672 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:04 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:16:04.672 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:04 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-22T05:16:04.672 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:04 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-22T05:16:04.672 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:04 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-22T05:16:04.672 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:04 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:16:04.672 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:04 smithi005 ceph-mon[24793]: Deploying daemon mgr.x on smithi186 2024-03-22T05:16:04.672 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:04 smithi005 ceph-mon[29523]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T05:16:04.672 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:04 smithi005 ceph-mon[29523]: from='client.34103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi005=y;smithi186=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:16:04.672 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:04 smithi005 ceph-mon[29523]: Saving service mgr spec with placement smithi005=y;smithi186=x;count:2 2024-03-22T05:16:04.672 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:04 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:16:04.672 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:04 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:16:04.672 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:04 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:16:04.672 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:04 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:16:04.673 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:04 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:16:04.673 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:04 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-22T05:16:04.673 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:04 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-22T05:16:04.673 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:04 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-22T05:16:04.673 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:04 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:16:04.673 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:04 smithi005 ceph-mon[29523]: Deploying daemon mgr.x on smithi186 2024-03-22T05:16:04.673 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records in 2024-03-22T05:16:04.674 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records out 2024-03-22T05:16:04.674 INFO:teuthology.orchestra.run.smithi005.stderr:512 bytes copied, 0.000112758 s, 4.5 MB/s 2024-03-22T05:16:04.675 DEBUG:teuthology.orchestra.run.smithi005:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-22T05:16:04.733 DEBUG:teuthology.orchestra.run.smithi005:> stat /dev/vg_nvme/lv_4 2024-03-22T05:16:04.790 INFO:teuthology.orchestra.run.smithi005.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-22T05:16:04.790 INFO:teuthology.orchestra.run.smithi005.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-22T05:16:04.790 INFO:teuthology.orchestra.run.smithi005.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-22T05:16:04.791 INFO:teuthology.orchestra.run.smithi005.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-22T05:16:04.791 INFO:teuthology.orchestra.run.smithi005.stdout:Context: system_u:object_r:device_t:s0 2024-03-22T05:16:04.791 INFO:teuthology.orchestra.run.smithi005.stdout:Access: 2024-03-22 05:15:39.455977922 +0000 2024-03-22T05:16:04.791 INFO:teuthology.orchestra.run.smithi005.stdout:Modify: 2024-03-22 05:13:21.545928831 +0000 2024-03-22T05:16:04.791 INFO:teuthology.orchestra.run.smithi005.stdout:Change: 2024-03-22 05:13:21.545928831 +0000 2024-03-22T05:16:04.791 INFO:teuthology.orchestra.run.smithi005.stdout: Birth: 2024-03-22 05:13:21.545928831 +0000 2024-03-22T05:16:04.791 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-22T05:16:04.860 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records in 2024-03-22T05:16:04.860 INFO:teuthology.orchestra.run.smithi005.stderr:1+0 records out 2024-03-22T05:16:04.860 INFO:teuthology.orchestra.run.smithi005.stderr:512 bytes copied, 0.00012174 s, 4.2 MB/s 2024-03-22T05:16:04.861 DEBUG:teuthology.orchestra.run.smithi005:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-22T05:16:04.920 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-03-22T05:16:04.920 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/scratch_devs of=/dev/stdout 2024-03-22T05:16:04.937 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-22T05:16:04.937 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_1 2024-03-22T05:16:04.994 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-22T05:16:04.994 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-22T05:16:04.995 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-22T05:16:04.995 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-22T05:16:04.995 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:device_t:s0 2024-03-22T05:16:04.995 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2024-03-22 05:15:48.625480426 +0000 2024-03-22T05:16:04.995 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2024-03-22 05:13:23.520556440 +0000 2024-03-22T05:16:04.995 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2024-03-22 05:13:23.520556440 +0000 2024-03-22T05:16:04.995 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: 2024-03-22 05:13:23.520556440 +0000 2024-03-22T05:16:04.995 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-22T05:16:05.062 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2024-03-22T05:16:05.062 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2024-03-22T05:16:05.062 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.000334116 s, 1.5 MB/s 2024-03-22T05:16:05.066 DEBUG:teuthology.orchestra.run.smithi186:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-22T05:16:05.123 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_2 2024-03-22T05:16:05.178 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-22T05:16:05.178 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-22T05:16:05.179 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-22T05:16:05.179 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-22T05:16:05.179 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:device_t:s0 2024-03-22T05:16:05.179 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2024-03-22 05:15:48.625480426 +0000 2024-03-22T05:16:05.179 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2024-03-22 05:13:23.859550147 +0000 2024-03-22T05:16:05.179 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2024-03-22 05:13:23.859550147 +0000 2024-03-22T05:16:05.179 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: 2024-03-22 05:13:23.859550147 +0000 2024-03-22T05:16:05.179 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-22T05:16:05.376 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2024-03-22T05:16:05.376 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2024-03-22T05:16:05.376 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.000202182 s, 2.5 MB/s 2024-03-22T05:16:05.377 DEBUG:teuthology.orchestra.run.smithi186:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-22T05:16:05.392 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_3 2024-03-22T05:16:05.452 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-22T05:16:05.452 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-22T05:16:05.452 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-22T05:16:05.452 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-22T05:16:05.452 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:device_t:s0 2024-03-22T05:16:05.453 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2024-03-22 05:16:05.337172805 +0000 2024-03-22T05:16:05.453 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2024-03-22 05:13:24.199543835 +0000 2024-03-22T05:16:05.453 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2024-03-22 05:13:24.199543835 +0000 2024-03-22T05:16:05.453 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: 2024-03-22 05:13:24.199543835 +0000 2024-03-22T05:16:05.453 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-22T05:16:05.548 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2024-03-22T05:16:05.548 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2024-03-22T05:16:05.548 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.000222848 s, 2.3 MB/s 2024-03-22T05:16:05.549 DEBUG:teuthology.orchestra.run.smithi186:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-22T05:16:05.564 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_4 2024-03-22T05:16:05.620 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-22T05:16:05.620 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-22T05:16:05.621 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-22T05:16:05.621 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-22T05:16:05.621 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:device_t:s0 2024-03-22T05:16:05.621 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2024-03-22 05:16:05.510169613 +0000 2024-03-22T05:16:05.621 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2024-03-22 05:13:24.567537004 +0000 2024-03-22T05:16:05.621 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2024-03-22 05:13:24.567537004 +0000 2024-03-22T05:16:05.621 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: 2024-03-22 05:13:24.567537004 +0000 2024-03-22T05:16:05.621 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-22T05:16:05.688 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2024-03-22T05:16:05.688 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2024-03-22T05:16:05.688 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.000273418 s, 1.9 MB/s 2024-03-22T05:16:05.689 DEBUG:teuthology.orchestra.run.smithi186:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-22T05:16:05.745 INFO:tasks.cephadm:Deploying osd.0 on smithi005 with /dev/vg_nvme/lv_4... 2024-03-22T05:16:05.745 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-22T05:16:06.044 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:16:06 smithi186 podman[26838]: 2024-03-22 05:16:06.029124878 +0000 UTC m=+0.364242962 container start febfc27d3c00bc86b6afb259b6a501db06db4e0a99882d1e3817b3cc8a2e57c4 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, io.k8s.display-name=CentOS Stream 8, vcs-type=git, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, release=754, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_BRANCH=HEAD) 2024-03-22T05:16:06.544 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:16:06 smithi186 ceph-mgr[26894]: -- 172.21.15.186:0/917835860 <== mon.2 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5646e7344340 con 0x5646e731c800 2024-03-22T05:16:06.544 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:16:06 smithi186 bash[26838]: febfc27d3c00bc86b6afb259b6a501db06db4e0a99882d1e3817b3cc8a2e57c4 2024-03-22T05:16:06.544 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:16:06 smithi186 systemd[1]: Started Ceph mgr.x for f59f36e2-e80a-11ee-95cd-87774f69a715. 2024-03-22T05:16:06.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:06 smithi005 ceph-mon[24793]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T05:16:06.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:06 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:16:06.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:06 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:16:06.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:06 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:16:06.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:06 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:16:06.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:06 smithi005 ceph-mon[29523]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T05:16:06.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:06 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:16:06.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:06 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:16:06.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:06 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:16:06.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:06 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:16:06.877 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:16:06 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: 2024-03-22T05:16:06.557+0000 7fab48b55000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-22T05:16:06.877 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:16:06 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: 2024-03-22T05:16:06.657+0000 7fab48b55000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-22T05:16:06.877 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:06 smithi186 ceph-mon[25720]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T05:16:06.877 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:06 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:16:06.877 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:06 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:16:06.877 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:06 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:16:06.877 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:06 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:16:07.544 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:16:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: 2024-03-22T05:16:07.280+0000 7fab48b55000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-22T05:16:07.544 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:16:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: 2024-03-22T05:16:07.370+0000 7fab48b55000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-22T05:16:07.544 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:16:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: 2024-03-22T05:16:07.453+0000 7fab48b55000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-22T05:16:07.778 INFO:teuthology.orchestra.run.smithi005.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-22T05:16:07.779 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-22T05:16:07.779 INFO:teuthology.orchestra.run.smithi005.stderr: stderr: 10+0 records in 2024-03-22T05:16:07.779 INFO:teuthology.orchestra.run.smithi005.stderr:10+0 records out 2024-03-22T05:16:07.779 INFO:teuthology.orchestra.run.smithi005.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00831106 s, 1.3 GB/s 2024-03-22T05:16:07.779 INFO:teuthology.orchestra.run.smithi005.stderr:--> Zapping successful for: 2024-03-22T05:16:07.878 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:16:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: 2024-03-22T05:16:07.609+0000 7fab48b55000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-22T05:16:07.878 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:16:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: 2024-03-22T05:16:07.688+0000 7fab48b55000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-22T05:16:07.878 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:16:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: 2024-03-22T05:16:07.796+0000 7fab48b55000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-22T05:16:07.878 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:16:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: 2024-03-22T05:16:07.878+0000 7fab48b55000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-22T05:16:08.221 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:16:08 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: 2024-03-22T05:16:08.039+0000 7fab48b55000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-22T05:16:08.221 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:16:08 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: 2024-03-22T05:16:08.124+0000 7fab48b55000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-22T05:16:08.511 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:08 smithi005 ceph-mon[29523]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T05:16:08.511 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:08 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:16:08.512 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:08 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:16:08.512 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:08 smithi005 ceph-mon[29523]: Reconfiguring mgr.y (unknown last config time)... 2024-03-22T05:16:08.512 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:08 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-22T05:16:08.512 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:08 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-22T05:16:08.512 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:08 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:16:08.512 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:08 smithi005 ceph-mon[29523]: Reconfiguring daemon mgr.y on smithi005 2024-03-22T05:16:08.512 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:08 smithi005 ceph-mon[24793]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T05:16:08.512 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:08 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:16:08.512 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:08 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:16:08.512 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:08 smithi005 ceph-mon[24793]: Reconfiguring mgr.y (unknown last config time)... 2024-03-22T05:16:08.512 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:08 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-22T05:16:08.512 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:08 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-22T05:16:08.512 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:08 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:16:08.513 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:08 smithi005 ceph-mon[24793]: Reconfiguring daemon mgr.y on smithi005 2024-03-22T05:16:08.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:08 smithi186 ceph-mon[25720]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T05:16:08.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:08 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:16:08.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:08 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:16:08.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:08 smithi186 ceph-mon[25720]: Reconfiguring mgr.y (unknown last config time)... 2024-03-22T05:16:08.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:08 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-22T05:16:08.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:08 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-22T05:16:08.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:08 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:16:08.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:08 smithi186 ceph-mon[25720]: Reconfiguring daemon mgr.y on smithi005 2024-03-22T05:16:09.157 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi005:vg_nvme/lv_4 2024-03-22T05:16:09.481 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:16:09 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: 2024-03-22T05:16:09.164+0000 7fab48b55000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-22T05:16:09.482 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:16:09 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: 2024-03-22T05:16:09.293+0000 7fab48b55000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-22T05:16:09.482 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:16:09 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: 2024-03-22T05:16:09.482+0000 7fab48b55000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-22T05:16:09.794 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:16:09 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: 2024-03-22T05:16:09.569+0000 7fab48b55000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-22T05:16:09.794 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:16:09 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: 2024-03-22T05:16:09.705+0000 7fab48b55000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-22T05:16:09.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:09 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:16:09.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:09 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:16:09.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:09 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:16:09.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:09 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:16:09.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:09 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:16:09.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:09 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-22T05:16:09.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:09 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-22T05:16:09.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:09 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:16:09.832 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:09 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:16:09.832 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:09 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:16:09.832 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:09 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:16:09.832 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:09 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:16:09.832 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:09 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:16:09.832 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:09 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-22T05:16:09.832 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:09 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-22T05:16:09.833 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:09 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:16:09.833 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:09 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:16:09.833 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:09 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:16:09.833 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:09 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:16:09.833 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:09 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:16:09.833 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:09 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:16:09.833 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:09 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-22T05:16:09.833 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:09 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-22T05:16:09.833 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:09 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:16:10.293 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:16:10 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: 2024-03-22T05:16:10.043+0000 7fab48b55000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-22T05:16:10.294 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:16:10 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: 2024-03-22T05:16:10.262+0000 7fab48b55000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-22T05:16:10.597 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:10 smithi005 ceph-mon[24793]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T05:16:10.597 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:10 smithi005 ceph-mon[29523]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T05:16:10.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:10 smithi186 ceph-mon[25720]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T05:16:10.794 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:16:10 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: 2024-03-22T05:16:10.345+0000 7fab48b55000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-22T05:16:10.794 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:16:10 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: 2024-03-22T05:16:10.515+0000 7fab48b55000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-22T05:16:11.674 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:11 smithi005 ceph-mon[24793]: from='client.14214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi005:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:16:11.675 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:11 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T05:16:11.675 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:11 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T05:16:11.675 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:11 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:16:11.675 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:11 smithi005 ceph-mon[29523]: from='client.14214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi005:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:16:11.675 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:11 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T05:16:11.675 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:11 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T05:16:11.675 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:11 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:16:11.725 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:16:11 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: 2024-03-22T05:16:11.434+0000 7fab48b55000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-22T05:16:11.725 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:16:11 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: 2024-03-22T05:16:11.725+0000 7fab48b55000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-22T05:16:11.725 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:11 smithi186 ceph-mon[25720]: from='client.14214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi005:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:16:11.725 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:11 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T05:16:11.725 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:11 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T05:16:11.725 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:11 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:16:12.044 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:16:11 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: 2024-03-22T05:16:11.818+0000 7fab48b55000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-22T05:16:12.044 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:16:11 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: 2024-03-22T05:16:11.965+0000 7fab48b55000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-22T05:16:12.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:12 smithi186 ceph-mon[25720]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T05:16:12.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:12 smithi186 ceph-mon[25720]: Standby manager daemon x started 2024-03-22T05:16:12.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:12 smithi186 ceph-mon[25720]: from='mgr.? 172.21.15.186:0/153075197' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-22T05:16:12.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:12 smithi186 ceph-mon[25720]: from='mgr.? 172.21.15.186:0/153075197' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-22T05:16:12.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:12 smithi186 ceph-mon[25720]: from='mgr.? 172.21.15.186:0/153075197' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-22T05:16:12.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:12 smithi186 ceph-mon[25720]: from='mgr.? 172.21.15.186:0/153075197' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-22T05:16:12.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:12 smithi005 ceph-mon[24793]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T05:16:12.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:12 smithi005 ceph-mon[24793]: Standby manager daemon x started 2024-03-22T05:16:12.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:12 smithi005 ceph-mon[24793]: from='mgr.? 172.21.15.186:0/153075197' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-22T05:16:12.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:12 smithi005 ceph-mon[24793]: from='mgr.? 172.21.15.186:0/153075197' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-22T05:16:12.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:12 smithi005 ceph-mon[24793]: from='mgr.? 172.21.15.186:0/153075197' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-22T05:16:12.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:12 smithi005 ceph-mon[24793]: from='mgr.? 172.21.15.186:0/153075197' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-22T05:16:12.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:12 smithi005 ceph-mon[29523]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T05:16:12.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:12 smithi005 ceph-mon[29523]: Standby manager daemon x started 2024-03-22T05:16:12.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:12 smithi005 ceph-mon[29523]: from='mgr.? 172.21.15.186:0/153075197' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-22T05:16:12.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:12 smithi005 ceph-mon[29523]: from='mgr.? 172.21.15.186:0/153075197' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-22T05:16:12.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:12 smithi005 ceph-mon[29523]: from='mgr.? 172.21.15.186:0/153075197' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-22T05:16:12.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:12 smithi005 ceph-mon[29523]: from='mgr.? 172.21.15.186:0/153075197' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-22T05:16:13.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:13 smithi186 ceph-mon[25720]: mgrmap e14: y(active, since 63s), standbys: x 2024-03-22T05:16:13.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:13 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-22T05:16:13.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:13 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/1976040891' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a0063741-df11-4674-9d0e-0ac5161bd1b8"}]: dispatch 2024-03-22T05:16:13.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:13 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/1976040891' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a0063741-df11-4674-9d0e-0ac5161bd1b8"}]': finished 2024-03-22T05:16:13.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:13 smithi186 ceph-mon[25720]: osdmap e5: 1 total, 0 up, 1 in 2024-03-22T05:16:13.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:13 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T05:16:13.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:13 smithi005 ceph-mon[24793]: mgrmap e14: y(active, since 63s), standbys: x 2024-03-22T05:16:13.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:13 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-22T05:16:13.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:13 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/1976040891' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a0063741-df11-4674-9d0e-0ac5161bd1b8"}]: dispatch 2024-03-22T05:16:13.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:13 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/1976040891' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a0063741-df11-4674-9d0e-0ac5161bd1b8"}]': finished 2024-03-22T05:16:13.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:13 smithi005 ceph-mon[24793]: osdmap e5: 1 total, 0 up, 1 in 2024-03-22T05:16:13.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:13 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T05:16:13.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:13 smithi005 ceph-mon[29523]: mgrmap e14: y(active, since 63s), standbys: x 2024-03-22T05:16:13.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:13 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-22T05:16:13.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:13 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/1976040891' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a0063741-df11-4674-9d0e-0ac5161bd1b8"}]: dispatch 2024-03-22T05:16:13.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:13 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/1976040891' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a0063741-df11-4674-9d0e-0ac5161bd1b8"}]': finished 2024-03-22T05:16:13.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:13 smithi005 ceph-mon[29523]: osdmap e5: 1 total, 0 up, 1 in 2024-03-22T05:16:13.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:13 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T05:16:14.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:14 smithi186 ceph-mon[25720]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T05:16:14.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:14 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/3662836841' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T05:16:14.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:14 smithi005 ceph-mon[29523]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T05:16:14.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:14 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/3662836841' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T05:16:14.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:14 smithi005 ceph-mon[24793]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T05:16:14.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:14 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/3662836841' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T05:16:16.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:16 smithi186 ceph-mon[25720]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T05:16:16.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:16 smithi005 ceph-mon[24793]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T05:16:16.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:16 smithi005 ceph-mon[29523]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T05:16:19.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:19 smithi005 ceph-mon[24793]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T05:16:19.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:19 smithi005 ceph-mon[29523]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T05:16:19.497 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:19 smithi186 ceph-mon[25720]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T05:16:20.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:20 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-22T05:16:20.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:20 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:16:20.354 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:20 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-22T05:16:20.355 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:20 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:16:20.355 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:20 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-22T05:16:20.355 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:20 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:16:21.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:21 smithi005 ceph-mon[24793]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T05:16:21.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:21 smithi005 ceph-mon[24793]: Deploying daemon osd.0 on smithi005 2024-03-22T05:16:21.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:21 smithi005 ceph-mon[29523]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T05:16:21.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:21 smithi005 ceph-mon[29523]: Deploying daemon osd.0 on smithi005 2024-03-22T05:16:21.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:21 smithi186 ceph-mon[25720]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T05:16:21.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:21 smithi186 ceph-mon[25720]: Deploying daemon osd.0 on smithi005 2024-03-22T05:16:23.351 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:23 smithi005 ceph-mon[24793]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T05:16:23.351 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:23 smithi005 ceph-mon[29523]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T05:16:23.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:23 smithi186 ceph-mon[25720]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T05:16:25.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:25 smithi005 ceph-mon[24793]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T05:16:25.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:25 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:16:25.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:25 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:16:25.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:25 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:16:25.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:25 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:16:25.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:25 smithi005 ceph-mon[29523]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T05:16:25.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:25 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:16:25.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:25 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:16:25.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:25 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:16:25.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:25 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:16:25.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:25 smithi186 ceph-mon[25720]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T05:16:25.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:25 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:16:25.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:25 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:16:25.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:25 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:16:25.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:25 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:16:27.309 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:27 smithi005 ceph-mon[24793]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T05:16:27.309 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:27 smithi005 ceph-mon[29523]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T05:16:27.362 INFO:teuthology.orchestra.run.smithi005.stdout:Created osd(s) 0 on host 'smithi005' 2024-03-22T05:16:27.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:27 smithi186 ceph-mon[25720]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T05:16:27.959 DEBUG:teuthology.orchestra.run.smithi005:osd.0> sudo journalctl -f -n 0 -u ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@osd.0.service 2024-03-22T05:16:27.962 INFO:tasks.cephadm:Deploying osd.1 on smithi005 with /dev/vg_nvme/lv_3... 2024-03-22T05:16:27.962 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-22T05:16:28.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:28 smithi005 ceph-mon[24793]: from='osd.0 [v2:172.21.15.5:6802/2350146241,v1:172.21.15.5:6803/2350146241]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-22T05:16:28.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:28 smithi005 ceph-mon[24793]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-22T05:16:28.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:28 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:16:28.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:28 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:16:28.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:28 smithi005 ceph-mon[29523]: from='osd.0 [v2:172.21.15.5:6802/2350146241,v1:172.21.15.5:6803/2350146241]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-22T05:16:28.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:28 smithi005 ceph-mon[29523]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-22T05:16:28.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:28 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:16:28.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:28 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:16:28.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:28 smithi186 ceph-mon[25720]: from='osd.0 [v2:172.21.15.5:6802/2350146241,v1:172.21.15.5:6803/2350146241]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-22T05:16:28.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:28 smithi186 ceph-mon[25720]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-22T05:16:28.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:28 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:16:28.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:28 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:16:29.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:29 smithi005 ceph-mon[24793]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T05:16:29.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:29 smithi005 ceph-mon[24793]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-22T05:16:29.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:29 smithi005 ceph-mon[24793]: osdmap e6: 1 total, 0 up, 1 in 2024-03-22T05:16:29.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:29 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T05:16:29.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:29 smithi005 ceph-mon[24793]: from='osd.0 [v2:172.21.15.5:6802/2350146241,v1:172.21.15.5:6803/2350146241]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-03-22T05:16:29.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:29 smithi005 ceph-mon[24793]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-03-22T05:16:29.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:29 smithi005 ceph-mon[29523]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T05:16:29.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:29 smithi005 ceph-mon[29523]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-22T05:16:29.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:29 smithi005 ceph-mon[29523]: osdmap e6: 1 total, 0 up, 1 in 2024-03-22T05:16:29.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:29 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T05:16:29.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:29 smithi005 ceph-mon[29523]: from='osd.0 [v2:172.21.15.5:6802/2350146241,v1:172.21.15.5:6803/2350146241]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-03-22T05:16:29.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:29 smithi005 ceph-mon[29523]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-03-22T05:16:29.204 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 22 05:16:29 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-0[32361]: 2024-03-22T05:16:29.095+0000 7f1293048700 -1 osd.0 0 waiting for initial osdmap 2024-03-22T05:16:29.204 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 22 05:16:29 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-0[32361]: 2024-03-22T05:16:29.097+0000 7f128d9df700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-22T05:16:29.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:29 smithi186 ceph-mon[25720]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T05:16:29.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:29 smithi186 ceph-mon[25720]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-22T05:16:29.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:29 smithi186 ceph-mon[25720]: osdmap e6: 1 total, 0 up, 1 in 2024-03-22T05:16:29.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:29 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T05:16:29.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:29 smithi186 ceph-mon[25720]: from='osd.0 [v2:172.21.15.5:6802/2350146241,v1:172.21.15.5:6803/2350146241]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-03-22T05:16:29.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:29 smithi186 ceph-mon[25720]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-03-22T05:16:30.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:30 smithi005 ceph-mon[24793]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]': finished 2024-03-22T05:16:30.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:30 smithi005 ceph-mon[24793]: osdmap e7: 1 total, 0 up, 1 in 2024-03-22T05:16:30.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:30 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T05:16:30.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:30 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T05:16:30.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:30 smithi005 ceph-mon[29523]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]': finished 2024-03-22T05:16:30.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:30 smithi005 ceph-mon[29523]: osdmap e7: 1 total, 0 up, 1 in 2024-03-22T05:16:30.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:30 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T05:16:30.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:30 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T05:16:30.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:30 smithi186 ceph-mon[25720]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]': finished 2024-03-22T05:16:30.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:30 smithi186 ceph-mon[25720]: osdmap e7: 1 total, 0 up, 1 in 2024-03-22T05:16:30.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:30 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T05:16:30.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:30 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T05:16:30.586 INFO:teuthology.orchestra.run.smithi005.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-22T05:16:30.586 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-22T05:16:30.586 INFO:teuthology.orchestra.run.smithi005.stderr: stderr: 10+0 records in 2024-03-22T05:16:30.586 INFO:teuthology.orchestra.run.smithi005.stderr:10+0 records out 2024-03-22T05:16:30.586 INFO:teuthology.orchestra.run.smithi005.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0105964 s, 990 MB/s 2024-03-22T05:16:30.586 INFO:teuthology.orchestra.run.smithi005.stderr:--> Zapping successful for: 2024-03-22T05:16:31.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:31 smithi005 ceph-mon[24793]: purged_snaps scrub starts 2024-03-22T05:16:31.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:31 smithi005 ceph-mon[24793]: purged_snaps scrub ok 2024-03-22T05:16:31.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:31 smithi005 ceph-mon[24793]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T05:16:31.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:31 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T05:16:31.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:31 smithi005 ceph-mon[24793]: osd.0 [v2:172.21.15.5:6802/2350146241,v1:172.21.15.5:6803/2350146241] boot 2024-03-22T05:16:31.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:31 smithi005 ceph-mon[24793]: osdmap e8: 1 total, 1 up, 1 in 2024-03-22T05:16:31.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:31 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T05:16:31.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:31 smithi005 ceph-mon[29523]: purged_snaps scrub starts 2024-03-22T05:16:31.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:31 smithi005 ceph-mon[29523]: purged_snaps scrub ok 2024-03-22T05:16:31.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:31 smithi005 ceph-mon[29523]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T05:16:31.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:31 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T05:16:31.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:31 smithi005 ceph-mon[29523]: osd.0 [v2:172.21.15.5:6802/2350146241,v1:172.21.15.5:6803/2350146241] boot 2024-03-22T05:16:31.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:31 smithi005 ceph-mon[29523]: osdmap e8: 1 total, 1 up, 1 in 2024-03-22T05:16:31.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:31 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T05:16:31.529 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi005:vg_nvme/lv_3 2024-03-22T05:16:31.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:31 smithi186 ceph-mon[25720]: purged_snaps scrub starts 2024-03-22T05:16:31.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:31 smithi186 ceph-mon[25720]: purged_snaps scrub ok 2024-03-22T05:16:31.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:31 smithi186 ceph-mon[25720]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-22T05:16:31.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:31 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T05:16:31.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:31 smithi186 ceph-mon[25720]: osd.0 [v2:172.21.15.5:6802/2350146241,v1:172.21.15.5:6803/2350146241] boot 2024-03-22T05:16:31.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:31 smithi186 ceph-mon[25720]: osdmap e8: 1 total, 1 up, 1 in 2024-03-22T05:16:31.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:31 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T05:16:32.403 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:32 smithi005 ceph-mon[24793]: osdmap e9: 1 total, 1 up, 1 in 2024-03-22T05:16:32.403 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:32 smithi005 ceph-mon[29523]: osdmap e9: 1 total, 1 up, 1 in 2024-03-22T05:16:32.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:32 smithi186 ceph-mon[25720]: osdmap e9: 1 total, 1 up, 1 in 2024-03-22T05:16:33.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:33 smithi186 ceph-mon[25720]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-22T05:16:33.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:33 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:16:33.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:33 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:16:33.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:33 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:16:33.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:33 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:16:33.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:33 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:16:33.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:33 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:16:33.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:33 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:16:33.325 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:33 smithi005 ceph-mon[29523]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-22T05:16:33.325 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:33 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:16:33.325 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:33 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:16:33.325 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:33 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:16:33.325 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:33 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:16:33.325 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:33 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:16:33.325 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:33 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:16:33.325 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:33 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:16:33.326 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:33 smithi005 ceph-mon[24793]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-22T05:16:33.326 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:33 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:16:33.326 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:33 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:16:33.326 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:33 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:16:33.326 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:33 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:16:33.326 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:33 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:16:33.326 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:33 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:16:33.326 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:33 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:16:34.359 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:34 smithi005 ceph-mon[24793]: Detected new or changed devices on smithi005 2024-03-22T05:16:34.359 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:34 smithi005 ceph-mon[24793]: Adjusting osd_memory_target on smithi005 to 16069M 2024-03-22T05:16:34.359 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:34 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:16:34.359 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:34 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T05:16:34.359 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:34 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T05:16:34.359 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:34 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:16:34.359 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:34 smithi005 ceph-mon[29523]: Detected new or changed devices on smithi005 2024-03-22T05:16:34.359 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:34 smithi005 ceph-mon[29523]: Adjusting osd_memory_target on smithi005 to 16069M 2024-03-22T05:16:34.359 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:34 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:16:34.360 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:34 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T05:16:34.360 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:34 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T05:16:34.360 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:34 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:16:34.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:34 smithi186 ceph-mon[25720]: Detected new or changed devices on smithi005 2024-03-22T05:16:34.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:34 smithi186 ceph-mon[25720]: Adjusting osd_memory_target on smithi005 to 16069M 2024-03-22T05:16:34.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:34 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:16:34.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:34 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T05:16:34.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:34 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T05:16:34.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:34 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:16:35.276 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:35 smithi005 ceph-mon[24793]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-22T05:16:35.276 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:35 smithi005 ceph-mon[24793]: from='client.14241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi005:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:16:35.276 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:35 smithi005 ceph-mon[29523]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-22T05:16:35.276 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:35 smithi005 ceph-mon[29523]: from='client.14241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi005:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:16:35.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:35 smithi186 ceph-mon[25720]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-22T05:16:35.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:35 smithi186 ceph-mon[25720]: from='client.14241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi005:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:16:36.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:36 smithi005 ceph-mon[24793]: pgmap v42: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-22T05:16:36.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:36 smithi005 ceph-mon[29523]: pgmap v42: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-22T05:16:36.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:36 smithi186 ceph-mon[25720]: pgmap v42: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-22T05:16:37.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:37 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/3501754696' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ef4e89fc-61a9-46c7-8357-2378e58055b3"}]: dispatch 2024-03-22T05:16:37.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:37 smithi005 ceph-mon[29523]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ef4e89fc-61a9-46c7-8357-2378e58055b3"}]: dispatch 2024-03-22T05:16:37.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:37 smithi005 ceph-mon[29523]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ef4e89fc-61a9-46c7-8357-2378e58055b3"}]': finished 2024-03-22T05:16:37.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:37 smithi005 ceph-mon[29523]: osdmap e10: 2 total, 1 up, 2 in 2024-03-22T05:16:37.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:37 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T05:16:37.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:37 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/3121150722' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T05:16:37.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:37 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/3501754696' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ef4e89fc-61a9-46c7-8357-2378e58055b3"}]: dispatch 2024-03-22T05:16:37.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:37 smithi005 ceph-mon[24793]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ef4e89fc-61a9-46c7-8357-2378e58055b3"}]: dispatch 2024-03-22T05:16:37.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:37 smithi005 ceph-mon[24793]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ef4e89fc-61a9-46c7-8357-2378e58055b3"}]': finished 2024-03-22T05:16:37.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:37 smithi005 ceph-mon[24793]: osdmap e10: 2 total, 1 up, 2 in 2024-03-22T05:16:37.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:37 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T05:16:37.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:37 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/3121150722' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T05:16:37.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:37 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/3501754696' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ef4e89fc-61a9-46c7-8357-2378e58055b3"}]: dispatch 2024-03-22T05:16:37.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:37 smithi186 ceph-mon[25720]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ef4e89fc-61a9-46c7-8357-2378e58055b3"}]: dispatch 2024-03-22T05:16:37.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:37 smithi186 ceph-mon[25720]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ef4e89fc-61a9-46c7-8357-2378e58055b3"}]': finished 2024-03-22T05:16:37.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:37 smithi186 ceph-mon[25720]: osdmap e10: 2 total, 1 up, 2 in 2024-03-22T05:16:37.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:37 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T05:16:37.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:37 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/3121150722' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T05:16:39.359 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:39 smithi005 ceph-mon[24793]: pgmap v44: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-22T05:16:39.359 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:39 smithi005 ceph-mon[29523]: pgmap v44: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-22T05:16:39.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:39 smithi186 ceph-mon[25720]: pgmap v44: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-22T05:16:41.277 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:41 smithi005 ceph-mon[24793]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-22T05:16:41.277 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:41 smithi005 ceph-mon[29523]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-22T05:16:41.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:41 smithi186 ceph-mon[25720]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-22T05:16:43.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:43 smithi005 ceph-mon[24793]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-22T05:16:43.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:43 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-22T05:16:43.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:43 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:16:43.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:43 smithi005 ceph-mon[29523]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-22T05:16:43.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:43 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-22T05:16:43.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:43 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:16:43.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:43 smithi186 ceph-mon[25720]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-22T05:16:43.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:43 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-22T05:16:43.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:43 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:16:44.198 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:44 smithi005 ceph-mon[29523]: Deploying daemon osd.1 on smithi005 2024-03-22T05:16:44.198 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:44 smithi005 ceph-mon[24793]: Deploying daemon osd.1 on smithi005 2024-03-22T05:16:44.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:44 smithi186 ceph-mon[25720]: Deploying daemon osd.1 on smithi005 2024-03-22T05:16:45.354 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:45 smithi005 ceph-mon[24793]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-22T05:16:45.355 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:45 smithi005 ceph-mon[29523]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-22T05:16:45.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:45 smithi186 ceph-mon[25720]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-22T05:16:47.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:47 smithi005 ceph-mon[24793]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-22T05:16:47.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:47 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:16:47.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:47 smithi005 ceph-mon[29523]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-22T05:16:47.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:47 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:16:47.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:47 smithi186 ceph-mon[25720]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-22T05:16:47.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:47 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:16:48.396 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:48 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:16:48.396 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:48 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:16:48.396 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:48 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:16:48.397 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:48 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:16:48.397 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:48 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:16:48.397 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:48 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:16:48.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:48 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:16:48.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:48 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:16:48.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:48 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:16:49.290 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:49 smithi005 ceph-mon[24793]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-22T05:16:49.290 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:49 smithi005 ceph-mon[29523]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-22T05:16:49.445 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:49 smithi186 ceph-mon[25720]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-22T05:16:50.101 INFO:teuthology.orchestra.run.smithi005.stdout:Created osd(s) 1 on host 'smithi005' 2024-03-22T05:16:50.731 DEBUG:teuthology.orchestra.run.smithi005:osd.1> sudo journalctl -f -n 0 -u ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@osd.1.service 2024-03-22T05:16:50.733 INFO:tasks.cephadm:Deploying osd.2 on smithi005 with /dev/vg_nvme/lv_2... 2024-03-22T05:16:50.733 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-22T05:16:51.418 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:51 smithi005 ceph-mon[29523]: pgmap v50: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-22T05:16:51.418 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:51 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:16:51.418 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:51 smithi005 ceph-mon[29523]: from='osd.1 [v2:172.21.15.5:6810/790214800,v1:172.21.15.5:6811/790214800]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-22T05:16:51.418 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:51 smithi005 ceph-mon[29523]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-22T05:16:51.418 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:51 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:16:51.418 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:51 smithi005 ceph-mon[24793]: pgmap v50: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-22T05:16:51.418 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:51 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:16:51.419 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:51 smithi005 ceph-mon[24793]: from='osd.1 [v2:172.21.15.5:6810/790214800,v1:172.21.15.5:6811/790214800]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-22T05:16:51.419 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:51 smithi005 ceph-mon[24793]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-22T05:16:51.419 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:51 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:16:51.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:51 smithi186 ceph-mon[25720]: pgmap v50: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-22T05:16:51.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:51 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:16:51.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:51 smithi186 ceph-mon[25720]: from='osd.1 [v2:172.21.15.5:6810/790214800,v1:172.21.15.5:6811/790214800]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-22T05:16:51.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:51 smithi186 ceph-mon[25720]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-22T05:16:51.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:51 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:16:52.280 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:52 smithi005 ceph-mon[24793]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-22T05:16:52.280 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:52 smithi005 ceph-mon[24793]: osdmap e11: 2 total, 1 up, 2 in 2024-03-22T05:16:52.280 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:52 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T05:16:52.280 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:52 smithi005 ceph-mon[24793]: from='osd.1 [v2:172.21.15.5:6810/790214800,v1:172.21.15.5:6811/790214800]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-03-22T05:16:52.280 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:52 smithi005 ceph-mon[24793]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-03-22T05:16:52.281 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:52 smithi005 ceph-mon[29523]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-22T05:16:52.281 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:52 smithi005 ceph-mon[29523]: osdmap e11: 2 total, 1 up, 2 in 2024-03-22T05:16:52.281 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:52 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T05:16:52.281 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:52 smithi005 ceph-mon[29523]: from='osd.1 [v2:172.21.15.5:6810/790214800,v1:172.21.15.5:6811/790214800]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-03-22T05:16:52.281 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:52 smithi005 ceph-mon[29523]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-03-22T05:16:52.281 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 22 05:16:52 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-1[35327]: 2024-03-22T05:16:52.111+0000 7fc77f134700 -1 osd.1 0 waiting for initial osdmap 2024-03-22T05:16:52.281 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 22 05:16:52 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-1[35327]: 2024-03-22T05:16:52.114+0000 7fc77bacf700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-22T05:16:52.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:52 smithi186 ceph-mon[25720]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-22T05:16:52.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:52 smithi186 ceph-mon[25720]: osdmap e11: 2 total, 1 up, 2 in 2024-03-22T05:16:52.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:52 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T05:16:52.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:52 smithi186 ceph-mon[25720]: from='osd.1 [v2:172.21.15.5:6810/790214800,v1:172.21.15.5:6811/790214800]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-03-22T05:16:52.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:52 smithi186 ceph-mon[25720]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-03-22T05:16:53.419 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:53 smithi005 ceph-mon[24793]: pgmap v52: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-22T05:16:53.419 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:53 smithi005 ceph-mon[24793]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]': finished 2024-03-22T05:16:53.419 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:53 smithi005 ceph-mon[24793]: osdmap e12: 2 total, 1 up, 2 in 2024-03-22T05:16:53.419 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:53 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T05:16:53.419 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:53 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T05:16:53.419 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:53 smithi005 ceph-mon[29523]: pgmap v52: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-22T05:16:53.419 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:53 smithi005 ceph-mon[29523]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]': finished 2024-03-22T05:16:53.419 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:53 smithi005 ceph-mon[29523]: osdmap e12: 2 total, 1 up, 2 in 2024-03-22T05:16:53.420 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:53 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T05:16:53.420 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:53 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T05:16:53.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:53 smithi186 ceph-mon[25720]: pgmap v52: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-22T05:16:53.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:53 smithi186 ceph-mon[25720]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]': finished 2024-03-22T05:16:53.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:53 smithi186 ceph-mon[25720]: osdmap e12: 2 total, 1 up, 2 in 2024-03-22T05:16:53.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:53 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T05:16:53.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:53 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T05:16:54.231 INFO:teuthology.orchestra.run.smithi005.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-22T05:16:54.231 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-22T05:16:54.231 INFO:teuthology.orchestra.run.smithi005.stderr: stderr: 10+0 records in 2024-03-22T05:16:54.232 INFO:teuthology.orchestra.run.smithi005.stderr:10+0 records out 2024-03-22T05:16:54.232 INFO:teuthology.orchestra.run.smithi005.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00940326 s, 1.1 GB/s 2024-03-22T05:16:54.232 INFO:teuthology.orchestra.run.smithi005.stderr:--> Zapping successful for: 2024-03-22T05:16:54.433 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:54 smithi005 ceph-mon[24793]: purged_snaps scrub starts 2024-03-22T05:16:54.433 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:54 smithi005 ceph-mon[24793]: purged_snaps scrub ok 2024-03-22T05:16:54.433 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:54 smithi005 ceph-mon[24793]: osd.1 [v2:172.21.15.5:6810/790214800,v1:172.21.15.5:6811/790214800] boot 2024-03-22T05:16:54.433 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:54 smithi005 ceph-mon[24793]: osdmap e13: 2 total, 2 up, 2 in 2024-03-22T05:16:54.433 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:54 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T05:16:54.434 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:54 smithi005 ceph-mon[24793]: pgmap v55: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-22T05:16:54.434 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:54 smithi005 ceph-mon[29523]: purged_snaps scrub starts 2024-03-22T05:16:54.434 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:54 smithi005 ceph-mon[29523]: purged_snaps scrub ok 2024-03-22T05:16:54.434 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:54 smithi005 ceph-mon[29523]: osd.1 [v2:172.21.15.5:6810/790214800,v1:172.21.15.5:6811/790214800] boot 2024-03-22T05:16:54.434 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:54 smithi005 ceph-mon[29523]: osdmap e13: 2 total, 2 up, 2 in 2024-03-22T05:16:54.434 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:54 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T05:16:54.434 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:54 smithi005 ceph-mon[29523]: pgmap v55: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-22T05:16:54.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:54 smithi186 ceph-mon[25720]: purged_snaps scrub starts 2024-03-22T05:16:54.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:54 smithi186 ceph-mon[25720]: purged_snaps scrub ok 2024-03-22T05:16:54.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:54 smithi186 ceph-mon[25720]: osd.1 [v2:172.21.15.5:6810/790214800,v1:172.21.15.5:6811/790214800] boot 2024-03-22T05:16:54.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:54 smithi186 ceph-mon[25720]: osdmap e13: 2 total, 2 up, 2 in 2024-03-22T05:16:54.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:54 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T05:16:54.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:54 smithi186 ceph-mon[25720]: pgmap v55: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-22T05:16:55.415 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:55 smithi005 ceph-mon[24793]: osdmap e14: 2 total, 2 up, 2 in 2024-03-22T05:16:55.415 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:55 smithi005 ceph-mon[24793]: Detected new or changed devices on smithi005 2024-03-22T05:16:55.415 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:55 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:16:55.415 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:55 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:16:55.415 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:55 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:16:55.415 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:55 smithi005 ceph-mon[24793]: Adjusting osd_memory_target on smithi005 to 8034M 2024-03-22T05:16:55.415 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:55 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:16:55.415 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:55 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:16:55.415 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:55 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:16:55.415 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:55 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:16:55.416 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:55 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:16:55.416 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:55 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:16:55.416 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:55 smithi005 ceph-mon[29523]: osdmap e14: 2 total, 2 up, 2 in 2024-03-22T05:16:55.416 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:55 smithi005 ceph-mon[29523]: Detected new or changed devices on smithi005 2024-03-22T05:16:55.416 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:55 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:16:55.416 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:55 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:16:55.416 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:55 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:16:55.416 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:55 smithi005 ceph-mon[29523]: Adjusting osd_memory_target on smithi005 to 8034M 2024-03-22T05:16:55.416 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:55 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:16:55.416 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:55 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:16:55.416 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:55 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:16:55.417 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:55 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:16:55.417 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:55 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:16:55.417 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:55 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:16:55.441 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi005:vg_nvme/lv_2 2024-03-22T05:16:55.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:55 smithi186 ceph-mon[25720]: osdmap e14: 2 total, 2 up, 2 in 2024-03-22T05:16:55.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:55 smithi186 ceph-mon[25720]: Detected new or changed devices on smithi005 2024-03-22T05:16:55.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:55 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:16:55.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:55 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:16:55.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:55 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:16:55.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:55 smithi186 ceph-mon[25720]: Adjusting osd_memory_target on smithi005 to 8034M 2024-03-22T05:16:55.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:55 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:16:55.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:55 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:16:55.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:55 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:16:55.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:55 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:16:55.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:55 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:16:55.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:55 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:16:56.404 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:56 smithi005 ceph-mon[24793]: pgmap v57: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-03-22T05:16:56.404 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:56 smithi005 ceph-mon[29523]: pgmap v57: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-03-22T05:16:56.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:56 smithi186 ceph-mon[25720]: pgmap v57: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-03-22T05:16:58.379 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:58 smithi005 ceph-mon[24793]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-22T05:16:58.379 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:58 smithi005 ceph-mon[24793]: from='client.14268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi005:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:16:58.380 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:58 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T05:16:58.380 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:58 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T05:16:58.380 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:16:58 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:16:58.380 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:58 smithi005 ceph-mon[29523]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-22T05:16:58.380 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:58 smithi005 ceph-mon[29523]: from='client.14268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi005:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:16:58.380 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:58 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T05:16:58.380 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:58 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T05:16:58.380 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:16:58 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:16:58.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:58 smithi186 ceph-mon[25720]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-22T05:16:58.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:58 smithi186 ceph-mon[25720]: from='client.14268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi005:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:16:58.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:58 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T05:16:58.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:58 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T05:16:58.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:16:58 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:17:00.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:00 smithi186 ceph-mon[25720]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-22T05:17:00.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:00 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/3538076044' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c1365858-72bc-4a3a-84cb-ac8f1beaa725"}]: dispatch 2024-03-22T05:17:00.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:00 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/3538076044' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c1365858-72bc-4a3a-84cb-ac8f1beaa725"}]': finished 2024-03-22T05:17:00.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:00 smithi186 ceph-mon[25720]: osdmap e15: 3 total, 2 up, 3 in 2024-03-22T05:17:00.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:00 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T05:17:00.616 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:00 smithi005 ceph-mon[24793]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-22T05:17:00.616 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:00 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/3538076044' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c1365858-72bc-4a3a-84cb-ac8f1beaa725"}]: dispatch 2024-03-22T05:17:00.616 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:00 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/3538076044' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c1365858-72bc-4a3a-84cb-ac8f1beaa725"}]': finished 2024-03-22T05:17:00.616 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:00 smithi005 ceph-mon[24793]: osdmap e15: 3 total, 2 up, 3 in 2024-03-22T05:17:00.616 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:00 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T05:17:00.617 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:00 smithi005 ceph-mon[29523]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-22T05:17:00.617 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:00 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/3538076044' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c1365858-72bc-4a3a-84cb-ac8f1beaa725"}]: dispatch 2024-03-22T05:17:00.617 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:00 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/3538076044' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c1365858-72bc-4a3a-84cb-ac8f1beaa725"}]': finished 2024-03-22T05:17:00.617 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:00 smithi005 ceph-mon[29523]: osdmap e15: 3 total, 2 up, 3 in 2024-03-22T05:17:00.617 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:00 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T05:17:01.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:01 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/1115642624' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T05:17:01.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:01 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/1115642624' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T05:17:01.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:01 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/1115642624' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T05:17:02.477 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:02 smithi005 ceph-mon[24793]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-22T05:17:02.477 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:02 smithi005 ceph-mon[29523]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-22T05:17:02.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:02 smithi186 ceph-mon[25720]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-22T05:17:05.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:05 smithi005 ceph-mon[24793]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-22T05:17:05.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:05 smithi005 ceph-mon[29523]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-22T05:17:05.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:05 smithi186 ceph-mon[25720]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-22T05:17:06.324 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:06 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-22T05:17:06.324 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:06 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:17:06.325 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:06 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-22T05:17:06.325 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:06 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:17:06.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:06 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-22T05:17:06.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:06 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:17:07.299 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:07 smithi005 ceph-mon[24793]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-22T05:17:07.300 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:07 smithi005 ceph-mon[24793]: Deploying daemon osd.2 on smithi005 2024-03-22T05:17:07.300 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:07 smithi005 ceph-mon[29523]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-22T05:17:07.300 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:07 smithi005 ceph-mon[29523]: Deploying daemon osd.2 on smithi005 2024-03-22T05:17:07.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:07 smithi186 ceph-mon[25720]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-22T05:17:07.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:07 smithi186 ceph-mon[25720]: Deploying daemon osd.2 on smithi005 2024-03-22T05:17:09.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:09 smithi005 ceph-mon[24793]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-22T05:17:09.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:09 smithi005 ceph-mon[29523]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-22T05:17:09.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:09 smithi186 ceph-mon[25720]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-22T05:17:10.371 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:10 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-22T05:17:10.371 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:10 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-22T05:17:10.372 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:10 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-22T05:17:10.372 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:10 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-22T05:17:10.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:10 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-22T05:17:10.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:10 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-22T05:17:11.318 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:11 smithi005 ceph-mon[24793]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-22T05:17:11.318 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:11 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:17:11.318 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:11 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:17:11.318 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:11 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:17:11.318 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:11 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:17:11.319 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:11 smithi005 ceph-mon[29523]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-22T05:17:11.319 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:11 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:17:11.319 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:11 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:17:11.319 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:11 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:17:11.319 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:11 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:17:11.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:11 smithi186 ceph-mon[25720]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-22T05:17:11.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:11 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:17:11.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:11 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:17:11.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:11 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:17:11.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:11 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:17:13.311 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:13 smithi005 ceph-mon[24793]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-22T05:17:13.312 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:13 smithi005 ceph-mon[29523]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-22T05:17:13.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:13 smithi186 ceph-mon[25720]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-22T05:17:14.228 INFO:teuthology.orchestra.run.smithi005.stdout:Created osd(s) 2 on host 'smithi005' 2024-03-22T05:17:14.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:14 smithi186 ceph-mon[25720]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-22T05:17:14.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:14 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:17:14.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:14 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:17:14.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:14 smithi186 ceph-mon[25720]: from='osd.2 [v2:172.21.15.5:6818/3354758722,v1:172.21.15.5:6819/3354758722]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-22T05:17:14.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:14 smithi186 ceph-mon[25720]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-22T05:17:14.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:14 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:17:14.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:14 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:17:14.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:14 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:17:14.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:14 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:17:14.835 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:14 smithi005 ceph-mon[29523]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-22T05:17:14.835 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:14 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:17:14.836 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:14 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:17:14.836 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:14 smithi005 ceph-mon[29523]: from='osd.2 [v2:172.21.15.5:6818/3354758722,v1:172.21.15.5:6819/3354758722]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-22T05:17:14.836 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:14 smithi005 ceph-mon[29523]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-22T05:17:14.836 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:14 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:17:14.836 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:14 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:17:14.836 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:14 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:17:14.836 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:14 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:17:14.836 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:14 smithi005 ceph-mon[24793]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-22T05:17:14.836 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:14 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:17:14.836 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:14 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:17:14.836 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:14 smithi005 ceph-mon[24793]: from='osd.2 [v2:172.21.15.5:6818/3354758722,v1:172.21.15.5:6819/3354758722]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-22T05:17:14.836 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:14 smithi005 ceph-mon[24793]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-22T05:17:14.836 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:14 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:17:14.837 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:14 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:17:14.837 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:14 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:17:14.837 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:14 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:17:14.858 DEBUG:teuthology.orchestra.run.smithi005:osd.2> sudo journalctl -f -n 0 -u ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@osd.2.service 2024-03-22T05:17:14.860 INFO:tasks.cephadm:Deploying osd.3 on smithi005 with /dev/vg_nvme/lv_1... 2024-03-22T05:17:14.861 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-22T05:17:15.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:15 smithi186 ceph-mon[25720]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-22T05:17:15.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:15 smithi186 ceph-mon[25720]: osdmap e16: 3 total, 2 up, 3 in 2024-03-22T05:17:15.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:15 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T05:17:15.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:15 smithi186 ceph-mon[25720]: from='osd.2 [v2:172.21.15.5:6818/3354758722,v1:172.21.15.5:6819/3354758722]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-03-22T05:17:15.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:15 smithi186 ceph-mon[25720]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-03-22T05:17:15.815 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:15 smithi005 ceph-mon[24793]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-22T05:17:15.815 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:15 smithi005 ceph-mon[24793]: osdmap e16: 3 total, 2 up, 3 in 2024-03-22T05:17:15.815 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:15 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T05:17:15.815 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:15 smithi005 ceph-mon[24793]: from='osd.2 [v2:172.21.15.5:6818/3354758722,v1:172.21.15.5:6819/3354758722]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-03-22T05:17:15.815 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:15 smithi005 ceph-mon[24793]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-03-22T05:17:15.815 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:15 smithi005 ceph-mon[29523]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-22T05:17:15.815 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:15 smithi005 ceph-mon[29523]: osdmap e16: 3 total, 2 up, 3 in 2024-03-22T05:17:15.816 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:15 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T05:17:15.816 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:15 smithi005 ceph-mon[29523]: from='osd.2 [v2:172.21.15.5:6818/3354758722,v1:172.21.15.5:6819/3354758722]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-03-22T05:17:15.816 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:15 smithi005 ceph-mon[29523]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-03-22T05:17:15.816 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 22 05:17:15 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-2[38242]: 2024-03-22T05:17:15.506+0000 7f85997fc700 -1 osd.2 0 waiting for initial osdmap 2024-03-22T05:17:15.816 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 22 05:17:15 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-2[38242]: 2024-03-22T05:17:15.509+0000 7f8596197700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-22T05:17:16.740 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:16 smithi005 ceph-mon[29523]: pgmap v69: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-22T05:17:16.740 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:16 smithi005 ceph-mon[29523]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]': finished 2024-03-22T05:17:16.740 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:16 smithi005 ceph-mon[29523]: osdmap e17: 3 total, 2 up, 3 in 2024-03-22T05:17:16.740 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:16 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T05:17:16.740 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:16 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T05:17:16.740 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:16 smithi005 ceph-mon[24793]: pgmap v69: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-22T05:17:16.741 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:16 smithi005 ceph-mon[24793]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]': finished 2024-03-22T05:17:16.741 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:16 smithi005 ceph-mon[24793]: osdmap e17: 3 total, 2 up, 3 in 2024-03-22T05:17:16.741 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:16 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T05:17:16.741 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:16 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T05:17:16.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:16 smithi186 ceph-mon[25720]: pgmap v69: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-22T05:17:16.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:16 smithi186 ceph-mon[25720]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]': finished 2024-03-22T05:17:16.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:16 smithi186 ceph-mon[25720]: osdmap e17: 3 total, 2 up, 3 in 2024-03-22T05:17:16.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:16 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T05:17:16.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:16 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T05:17:17.603 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:17 smithi005 ceph-mon[24793]: purged_snaps scrub starts 2024-03-22T05:17:17.603 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:17 smithi005 ceph-mon[24793]: purged_snaps scrub ok 2024-03-22T05:17:17.603 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:17 smithi005 ceph-mon[24793]: osd.2 [v2:172.21.15.5:6818/3354758722,v1:172.21.15.5:6819/3354758722] boot 2024-03-22T05:17:17.603 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:17 smithi005 ceph-mon[24793]: osdmap e18: 3 total, 3 up, 3 in 2024-03-22T05:17:17.603 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:17 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T05:17:17.603 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:17 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-03-22T05:17:17.603 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:17 smithi005 ceph-mon[29523]: purged_snaps scrub starts 2024-03-22T05:17:17.604 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:17 smithi005 ceph-mon[29523]: purged_snaps scrub ok 2024-03-22T05:17:17.604 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:17 smithi005 ceph-mon[29523]: osd.2 [v2:172.21.15.5:6818/3354758722,v1:172.21.15.5:6819/3354758722] boot 2024-03-22T05:17:17.604 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:17 smithi005 ceph-mon[29523]: osdmap e18: 3 total, 3 up, 3 in 2024-03-22T05:17:17.604 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:17 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T05:17:17.604 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:17 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-03-22T05:17:17.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:17 smithi186 ceph-mon[25720]: purged_snaps scrub starts 2024-03-22T05:17:17.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:17 smithi186 ceph-mon[25720]: purged_snaps scrub ok 2024-03-22T05:17:17.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:17 smithi186 ceph-mon[25720]: osd.2 [v2:172.21.15.5:6818/3354758722,v1:172.21.15.5:6819/3354758722] boot 2024-03-22T05:17:17.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:17 smithi186 ceph-mon[25720]: osdmap e18: 3 total, 3 up, 3 in 2024-03-22T05:17:17.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:17 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T05:17:17.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:17 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-03-22T05:17:18.771 INFO:teuthology.orchestra.run.smithi005.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-22T05:17:18.771 INFO:teuthology.orchestra.run.smithi005.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-22T05:17:18.771 INFO:teuthology.orchestra.run.smithi005.stderr: stderr: 10+0 records in 2024-03-22T05:17:18.771 INFO:teuthology.orchestra.run.smithi005.stderr:10+0 records out 2024-03-22T05:17:18.771 INFO:teuthology.orchestra.run.smithi005.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00825317 s, 1.3 GB/s 2024-03-22T05:17:18.771 INFO:teuthology.orchestra.run.smithi005.stderr:--> Zapping successful for: 2024-03-22T05:17:18.771 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:18 smithi005 ceph-mon[24793]: pgmap v72: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-22T05:17:18.772 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:18 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-03-22T05:17:18.772 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:18 smithi005 ceph-mon[24793]: osdmap e19: 3 total, 3 up, 3 in 2024-03-22T05:17:18.772 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:18 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-22T05:17:18.772 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:18 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:17:18.772 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:18 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:17:18.772 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:18 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:17:18.772 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:18 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:17:18.772 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:18 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:17:18.772 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:18 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:17:18.772 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:18 smithi005 ceph-mon[29523]: pgmap v72: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-22T05:17:18.772 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:18 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-03-22T05:17:18.772 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:18 smithi005 ceph-mon[29523]: osdmap e19: 3 total, 3 up, 3 in 2024-03-22T05:17:18.773 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:18 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-22T05:17:18.773 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:18 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:17:18.773 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:18 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:17:18.773 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:18 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:17:18.773 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:18 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:17:18.773 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:18 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:17:18.773 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:18 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:17:18.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:18 smithi186 ceph-mon[25720]: pgmap v72: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-22T05:17:18.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:18 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-03-22T05:17:18.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:18 smithi186 ceph-mon[25720]: osdmap e19: 3 total, 3 up, 3 in 2024-03-22T05:17:18.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:18 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-22T05:17:18.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:18 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:17:18.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:18 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:17:18.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:18 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:17:18.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:18 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:17:18.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:18 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:17:18.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:18 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:17:19.403 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi005:vg_nvme/lv_1 2024-03-22T05:17:19.665 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 22 05:17:19 smithi005 sudo[39463]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-22T05:17:19.665 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 22 05:17:19 smithi005 sudo[39463]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-22T05:17:19.666 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 22 05:17:19 smithi005 sudo[39463]: pam_unix(sudo:session): session closed for user root 2024-03-22T05:17:19.666 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:19 smithi005 ceph-mon[24793]: Detected new or changed devices on smithi005 2024-03-22T05:17:19.666 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:19 smithi005 ceph-mon[24793]: Adjusting osd_memory_target on smithi005 to 5356M 2024-03-22T05:17:19.666 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:19 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-22T05:17:19.666 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:19 smithi005 ceph-mon[24793]: osdmap e20: 3 total, 3 up, 3 in 2024-03-22T05:17:19.666 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:19 smithi005 ceph-mon[29523]: Detected new or changed devices on smithi005 2024-03-22T05:17:19.666 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:19 smithi005 ceph-mon[29523]: Adjusting osd_memory_target on smithi005 to 5356M 2024-03-22T05:17:19.667 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:19 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-22T05:17:19.667 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:19 smithi005 ceph-mon[29523]: osdmap e20: 3 total, 3 up, 3 in 2024-03-22T05:17:19.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:19 smithi186 ceph-mon[25720]: Detected new or changed devices on smithi005 2024-03-22T05:17:19.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:19 smithi186 ceph-mon[25720]: Adjusting osd_memory_target on smithi005 to 5356M 2024-03-22T05:17:19.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:19 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-22T05:17:19.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:19 smithi186 ceph-mon[25720]: osdmap e20: 3 total, 3 up, 3 in 2024-03-22T05:17:19.917 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 22 05:17:19 smithi005 sudo[39512]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-22T05:17:19.917 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 22 05:17:19 smithi005 sudo[39512]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-22T05:17:19.918 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 22 05:17:19 smithi005 sudo[39512]: pam_unix(sudo:session): session closed for user root 2024-03-22T05:17:20.202 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 22 05:17:19 smithi005 sudo[39532]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-22T05:17:20.202 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 22 05:17:19 smithi005 sudo[39532]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-22T05:17:20.202 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 22 05:17:19 smithi005 sudo[39532]: pam_unix(sudo:session): session closed for user root 2024-03-22T05:17:20.471 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 22 05:17:20 smithi005 sudo[39535]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-22T05:17:20.472 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 22 05:17:20 smithi005 sudo[39535]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-22T05:17:20.472 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 22 05:17:20 smithi005 sudo[39535]: pam_unix(sudo:session): session closed for user root 2024-03-22T05:17:20.723 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:20 smithi005 ceph-mon[24793]: pgmap v75: 1 pgs: 1 active+clean; 449 KiB data, 15 MiB used, 268 GiB / 268 GiB avail 2024-03-22T05:17:20.724 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:20 smithi005 ceph-mon[24793]: osdmap e21: 3 total, 3 up, 3 in 2024-03-22T05:17:20.724 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:20 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:17:20.724 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 22 05:17:20 smithi005 sudo[39541]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-22T05:17:20.724 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 22 05:17:20 smithi005 sudo[39541]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-22T05:17:20.724 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 22 05:17:20 smithi005 sudo[39541]: pam_unix(sudo:session): session closed for user root 2024-03-22T05:17:20.724 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:20 smithi005 ceph-mon[29523]: pgmap v75: 1 pgs: 1 active+clean; 449 KiB data, 15 MiB used, 268 GiB / 268 GiB avail 2024-03-22T05:17:20.724 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:20 smithi005 ceph-mon[29523]: osdmap e21: 3 total, 3 up, 3 in 2024-03-22T05:17:20.724 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:20 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:17:20.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:20 smithi186 ceph-mon[25720]: pgmap v75: 1 pgs: 1 active+clean; 449 KiB data, 15 MiB used, 268 GiB / 268 GiB avail 2024-03-22T05:17:20.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:20 smithi186 ceph-mon[25720]: osdmap e21: 3 total, 3 up, 3 in 2024-03-22T05:17:20.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:20 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:17:20.978 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 22 05:17:20 smithi005 sudo[39562]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-22T05:17:20.978 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 22 05:17:20 smithi005 sudo[39562]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-22T05:17:20.979 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 22 05:17:20 smithi005 sudo[39562]: pam_unix(sudo:session): session closed for user root 2024-03-22T05:17:21.273 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:21 smithi005 sudo[39571]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-22T05:17:21.274 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:21 smithi005 sudo[39571]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-22T05:17:21.274 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:21 smithi005 sudo[39571]: pam_unix(sudo:session): session closed for user root 2024-03-22T05:17:21.530 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:21 smithi005 sudo[39594]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-22T05:17:21.530 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:21 smithi005 sudo[39594]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-22T05:17:21.530 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:21 smithi005 sudo[39594]: pam_unix(sudo:session): session closed for user root 2024-03-22T05:17:21.784 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:21 smithi005 sudo[39655]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-22T05:17:21.785 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:21 smithi005 sudo[39655]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-22T05:17:21.785 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:21 smithi005 sudo[39655]: pam_unix(sudo:session): session closed for user root 2024-03-22T05:17:22.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:21 smithi005 sudo[39684]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-22T05:17:22.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:21 smithi005 sudo[39684]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-22T05:17:22.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:21 smithi005 sudo[39684]: pam_unix(sudo:session): session closed for user root 2024-03-22T05:17:22.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:22 smithi186 sudo[27219]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-22T05:17:22.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:22 smithi186 sudo[27219]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-22T05:17:22.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:22 smithi186 sudo[27219]: pam_unix(sudo:session): session closed for user root 2024-03-22T05:17:22.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:22 smithi186 sudo[27222]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-22T05:17:22.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:22 smithi186 sudo[27222]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-22T05:17:22.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:22 smithi186 sudo[27222]: pam_unix(sudo:session): session closed for user root 2024-03-22T05:17:22.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:22 smithi186 ceph-mon[25720]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-22T05:17:22.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:22 smithi186 ceph-mon[25720]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-22T05:17:22.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:22 smithi186 ceph-mon[25720]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-22T05:17:22.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:22 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-22T05:17:22.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:22 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T05:17:22.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:22 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T05:17:22.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:22 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T05:17:22.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:22 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-22T05:17:22.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:22 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T05:17:22.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:22 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T05:17:22.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:22 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T05:17:22.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:22 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:17:22.642 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:22 smithi005 ceph-mon[29523]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-22T05:17:22.642 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:22 smithi005 ceph-mon[29523]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-22T05:17:22.642 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:22 smithi005 ceph-mon[29523]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-22T05:17:22.642 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:22 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-22T05:17:22.642 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:22 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T05:17:22.643 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:22 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T05:17:22.643 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:22 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T05:17:22.643 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:22 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-22T05:17:22.643 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:22 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T05:17:22.643 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:22 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T05:17:22.643 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:22 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T05:17:22.643 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:22 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:17:22.643 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:22 smithi005 ceph-mon[24793]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-22T05:17:22.643 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:22 smithi005 ceph-mon[24793]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-22T05:17:22.643 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:22 smithi005 ceph-mon[24793]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-22T05:17:22.643 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:22 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-22T05:17:22.643 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:22 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T05:17:22.644 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:22 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T05:17:22.644 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:22 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T05:17:22.644 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:22 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-22T05:17:22.644 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:22 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T05:17:22.644 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:22 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T05:17:22.644 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:22 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T05:17:22.644 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:22 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:17:23.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:23 smithi005 ceph-mon[29523]: pgmap v77: 1 pgs: 1 active+clean; 449 KiB data, 16 MiB used, 268 GiB / 268 GiB avail 2024-03-22T05:17:23.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:23 smithi005 ceph-mon[29523]: from='client.24205 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi005:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:17:23.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:23 smithi005 ceph-mon[29523]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-22T05:17:23.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:23 smithi005 ceph-mon[29523]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-22T05:17:23.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:23 smithi005 ceph-mon[29523]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-22T05:17:23.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:23 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-22T05:17:23.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:23 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T05:17:23.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:23 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T05:17:23.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:23 smithi005 ceph-mon[29523]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-22T05:17:23.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:23 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:17:23.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:23 smithi005 ceph-mon[24793]: pgmap v77: 1 pgs: 1 active+clean; 449 KiB data, 16 MiB used, 268 GiB / 268 GiB avail 2024-03-22T05:17:23.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:23 smithi005 ceph-mon[24793]: from='client.24205 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi005:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:17:23.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:23 smithi005 ceph-mon[24793]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-22T05:17:23.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:23 smithi005 ceph-mon[24793]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-22T05:17:23.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:23 smithi005 ceph-mon[24793]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-22T05:17:23.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:23 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-22T05:17:23.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:23 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T05:17:23.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:23 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T05:17:23.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:23 smithi005 ceph-mon[24793]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-22T05:17:23.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:23 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:17:23.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:23 smithi186 ceph-mon[25720]: pgmap v77: 1 pgs: 1 active+clean; 449 KiB data, 16 MiB used, 268 GiB / 268 GiB avail 2024-03-22T05:17:23.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:23 smithi186 ceph-mon[25720]: from='client.24205 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi005:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:17:23.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:23 smithi186 ceph-mon[25720]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-22T05:17:23.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:23 smithi186 ceph-mon[25720]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-22T05:17:23.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:23 smithi186 ceph-mon[25720]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-22T05:17:23.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:23 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-22T05:17:23.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:23 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T05:17:23.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:23 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T05:17:23.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:23 smithi186 ceph-mon[25720]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-22T05:17:23.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:23 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:17:24.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:24 smithi005 ceph-mon[24793]: pgmap v78: 1 pgs: 1 active+clean; 449 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2024-03-22T05:17:24.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:24 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/3884159936' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ae0fa9b1-b54d-4c9a-b5d0-33ae91dcf98d"}]: dispatch 2024-03-22T05:17:24.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:24 smithi005 ceph-mon[24793]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ae0fa9b1-b54d-4c9a-b5d0-33ae91dcf98d"}]: dispatch 2024-03-22T05:17:24.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:24 smithi005 ceph-mon[24793]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ae0fa9b1-b54d-4c9a-b5d0-33ae91dcf98d"}]': finished 2024-03-22T05:17:24.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:24 smithi005 ceph-mon[24793]: osdmap e22: 4 total, 3 up, 4 in 2024-03-22T05:17:24.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:24 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T05:17:24.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:24 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/2584284362' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T05:17:24.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:24 smithi005 ceph-mon[29523]: pgmap v78: 1 pgs: 1 active+clean; 449 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2024-03-22T05:17:24.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:24 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/3884159936' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ae0fa9b1-b54d-4c9a-b5d0-33ae91dcf98d"}]: dispatch 2024-03-22T05:17:24.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:24 smithi005 ceph-mon[29523]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ae0fa9b1-b54d-4c9a-b5d0-33ae91dcf98d"}]: dispatch 2024-03-22T05:17:24.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:24 smithi005 ceph-mon[29523]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ae0fa9b1-b54d-4c9a-b5d0-33ae91dcf98d"}]': finished 2024-03-22T05:17:24.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:24 smithi005 ceph-mon[29523]: osdmap e22: 4 total, 3 up, 4 in 2024-03-22T05:17:24.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:24 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T05:17:24.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:24 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/2584284362' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T05:17:24.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:24 smithi186 ceph-mon[25720]: pgmap v78: 1 pgs: 1 active+clean; 449 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2024-03-22T05:17:24.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:24 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/3884159936' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ae0fa9b1-b54d-4c9a-b5d0-33ae91dcf98d"}]: dispatch 2024-03-22T05:17:24.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:24 smithi186 ceph-mon[25720]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ae0fa9b1-b54d-4c9a-b5d0-33ae91dcf98d"}]: dispatch 2024-03-22T05:17:24.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:24 smithi186 ceph-mon[25720]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ae0fa9b1-b54d-4c9a-b5d0-33ae91dcf98d"}]': finished 2024-03-22T05:17:24.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:24 smithi186 ceph-mon[25720]: osdmap e22: 4 total, 3 up, 4 in 2024-03-22T05:17:24.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:24 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T05:17:24.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:24 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/2584284362' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T05:17:26.660 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:26 smithi005 ceph-mon[24793]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 6.5 KiB/s rd, 261 KiB/s wr, 19 op/s 2024-03-22T05:17:26.660 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:26 smithi005 ceph-mon[29523]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 6.5 KiB/s rd, 261 KiB/s wr, 19 op/s 2024-03-22T05:17:26.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:26 smithi186 ceph-mon[25720]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 6.5 KiB/s rd, 261 KiB/s wr, 19 op/s 2024-03-22T05:17:29.289 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:29 smithi005 ceph-mon[29523]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 4.9 KiB/s rd, 196 KiB/s wr, 14 op/s 2024-03-22T05:17:29.289 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:29 smithi005 ceph-mon[24793]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 4.9 KiB/s rd, 196 KiB/s wr, 14 op/s 2024-03-22T05:17:29.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:29 smithi186 ceph-mon[25720]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 4.9 KiB/s rd, 196 KiB/s wr, 14 op/s 2024-03-22T05:17:30.384 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:30 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-22T05:17:30.384 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:30 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:17:30.385 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:30 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-22T05:17:30.385 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:30 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:17:30.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:30 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-22T05:17:30.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:30 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:17:31.313 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:31 smithi005 ceph-mon[24793]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 4.0 KiB/s rd, 163 KiB/s wr, 12 op/s 2024-03-22T05:17:31.313 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:31 smithi005 ceph-mon[24793]: Deploying daemon osd.3 on smithi005 2024-03-22T05:17:31.313 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:31 smithi005 ceph-mon[29523]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 4.0 KiB/s rd, 163 KiB/s wr, 12 op/s 2024-03-22T05:17:31.313 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:31 smithi005 ceph-mon[29523]: Deploying daemon osd.3 on smithi005 2024-03-22T05:17:31.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:31 smithi186 ceph-mon[25720]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 4.0 KiB/s rd, 163 KiB/s wr, 12 op/s 2024-03-22T05:17:31.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:31 smithi186 ceph-mon[25720]: Deploying daemon osd.3 on smithi005 2024-03-22T05:17:33.172 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:33 smithi005 ceph-mon[24793]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 3.9 KiB/s rd, 157 KiB/s wr, 11 op/s 2024-03-22T05:17:33.172 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:33 smithi005 ceph-mon[29523]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 3.9 KiB/s rd, 157 KiB/s wr, 11 op/s 2024-03-22T05:17:33.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:33 smithi186 ceph-mon[25720]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 3.9 KiB/s rd, 157 KiB/s wr, 11 op/s 2024-03-22T05:17:35.399 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:35 smithi005 ceph-mon[24793]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 3.9 KiB/s rd, 157 KiB/s wr, 11 op/s 2024-03-22T05:17:35.399 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:35 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:17:35.399 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:35 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:17:35.399 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:35 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:17:35.399 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:35 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:17:35.399 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:35 smithi005 ceph-mon[29523]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 3.9 KiB/s rd, 157 KiB/s wr, 11 op/s 2024-03-22T05:17:35.399 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:35 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:17:35.399 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:35 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:17:35.399 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:35 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:17:35.399 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:35 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:17:35.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:35 smithi186 ceph-mon[25720]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 3.9 KiB/s rd, 157 KiB/s wr, 11 op/s 2024-03-22T05:17:35.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:35 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:17:35.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:35 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:17:35.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:35 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:17:35.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:35 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:17:37.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:37 smithi005 ceph-mon[24793]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 3.3 KiB/s rd, 135 KiB/s wr, 10 op/s 2024-03-22T05:17:37.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:37 smithi005 ceph-mon[29523]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 3.3 KiB/s rd, 135 KiB/s wr, 10 op/s 2024-03-22T05:17:37.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:37 smithi186 ceph-mon[25720]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 3.3 KiB/s rd, 135 KiB/s wr, 10 op/s 2024-03-22T05:17:37.776 INFO:teuthology.orchestra.run.smithi005.stdout:Created osd(s) 3 on host 'smithi005' 2024-03-22T05:17:38.330 DEBUG:teuthology.orchestra.run.smithi005:osd.3> sudo journalctl -f -n 0 -u ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@osd.3.service 2024-03-22T05:17:38.332 INFO:tasks.cephadm:Deploying osd.4 on smithi186 with /dev/vg_nvme/lv_4... 2024-03-22T05:17:38.332 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-22T05:17:38.867 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:38 smithi005 ceph-mon[29523]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-22T05:17:38.867 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:38 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:17:38.867 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:38 smithi005 ceph-mon[29523]: from='osd.3 [v2:172.21.15.5:6826/3829723611,v1:172.21.15.5:6827/3829723611]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-22T05:17:38.867 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:38 smithi005 ceph-mon[29523]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-22T05:17:38.867 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:38 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:17:38.867 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:38 smithi005 ceph-mon[24793]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-22T05:17:38.867 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:38 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:17:38.867 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:38 smithi005 ceph-mon[24793]: from='osd.3 [v2:172.21.15.5:6826/3829723611,v1:172.21.15.5:6827/3829723611]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-22T05:17:38.868 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:38 smithi005 ceph-mon[24793]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-22T05:17:38.868 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:38 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:17:38.894 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:38 smithi186 ceph-mon[25720]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-22T05:17:38.894 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:38 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:17:38.895 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:38 smithi186 ceph-mon[25720]: from='osd.3 [v2:172.21.15.5:6826/3829723611,v1:172.21.15.5:6827/3829723611]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-22T05:17:38.895 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:38 smithi186 ceph-mon[25720]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-22T05:17:38.895 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:38 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:17:39.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:39 smithi005 ceph-mon[29523]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-22T05:17:39.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:39 smithi005 ceph-mon[29523]: osdmap e23: 4 total, 3 up, 4 in 2024-03-22T05:17:39.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:39 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T05:17:39.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:39 smithi005 ceph-mon[29523]: from='osd.3 [v2:172.21.15.5:6826/3829723611,v1:172.21.15.5:6827/3829723611]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-03-22T05:17:39.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:39 smithi005 ceph-mon[29523]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-03-22T05:17:39.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:39 smithi005 ceph-mon[24793]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-22T05:17:39.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:39 smithi005 ceph-mon[24793]: osdmap e23: 4 total, 3 up, 4 in 2024-03-22T05:17:39.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:39 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T05:17:39.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:39 smithi005 ceph-mon[24793]: from='osd.3 [v2:172.21.15.5:6826/3829723611,v1:172.21.15.5:6827/3829723611]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-03-22T05:17:39.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:39 smithi005 ceph-mon[24793]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-03-22T05:17:39.953 INFO:journalctl@ceph.osd.3.smithi005.stdout:Mar 22 05:17:39 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-3[41223]: 2024-03-22T05:17:39.785+0000 7f3811d49700 -1 osd.3 0 waiting for initial osdmap 2024-03-22T05:17:39.953 INFO:journalctl@ceph.osd.3.smithi005.stdout:Mar 22 05:17:39 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-3[41223]: 2024-03-22T05:17:39.789+0000 7f380c6e0700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-22T05:17:40.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:39 smithi186 ceph-mon[25720]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-22T05:17:40.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:39 smithi186 ceph-mon[25720]: osdmap e23: 4 total, 3 up, 4 in 2024-03-22T05:17:40.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:39 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T05:17:40.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:39 smithi186 ceph-mon[25720]: from='osd.3 [v2:172.21.15.5:6826/3829723611,v1:172.21.15.5:6827/3829723611]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-03-22T05:17:40.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:39 smithi186 ceph-mon[25720]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-03-22T05:17:40.293 INFO:teuthology.orchestra.run.smithi186.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-22T05:17:40.293 INFO:teuthology.orchestra.run.smithi186.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-22T05:17:40.293 INFO:teuthology.orchestra.run.smithi186.stderr: stderr: 10+0 records in 2024-03-22T05:17:40.293 INFO:teuthology.orchestra.run.smithi186.stderr:10+0 records out 2024-03-22T05:17:40.293 INFO:teuthology.orchestra.run.smithi186.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0133838 s, 783 MB/s 2024-03-22T05:17:40.293 INFO:teuthology.orchestra.run.smithi186.stderr:--> Zapping successful for: 2024-03-22T05:17:40.915 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi186:vg_nvme/lv_4 2024-03-22T05:17:40.940 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:40 smithi186 ceph-mon[25720]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-22T05:17:40.940 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:40 smithi186 ceph-mon[25720]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]': finished 2024-03-22T05:17:40.941 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:40 smithi186 ceph-mon[25720]: osdmap e24: 4 total, 3 up, 4 in 2024-03-22T05:17:40.941 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:40 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T05:17:40.941 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:40 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T05:17:41.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:40 smithi005 ceph-mon[24793]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-22T05:17:41.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:40 smithi005 ceph-mon[24793]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]': finished 2024-03-22T05:17:41.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:40 smithi005 ceph-mon[24793]: osdmap e24: 4 total, 3 up, 4 in 2024-03-22T05:17:41.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:40 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T05:17:41.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:40 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T05:17:41.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:40 smithi005 ceph-mon[29523]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-22T05:17:41.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:40 smithi005 ceph-mon[29523]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]': finished 2024-03-22T05:17:41.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:40 smithi005 ceph-mon[29523]: osdmap e24: 4 total, 3 up, 4 in 2024-03-22T05:17:41.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:40 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T05:17:41.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:40 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T05:17:41.878 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:41 smithi005 ceph-mon[29523]: purged_snaps scrub starts 2024-03-22T05:17:41.878 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:41 smithi005 ceph-mon[29523]: purged_snaps scrub ok 2024-03-22T05:17:41.879 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:41 smithi005 ceph-mon[29523]: osd.3 [v2:172.21.15.5:6826/3829723611,v1:172.21.15.5:6827/3829723611] boot 2024-03-22T05:17:41.879 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:41 smithi005 ceph-mon[29523]: osdmap e25: 4 total, 4 up, 4 in 2024-03-22T05:17:41.879 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:41 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T05:17:41.879 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:41 smithi005 ceph-mon[24793]: purged_snaps scrub starts 2024-03-22T05:17:41.879 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:41 smithi005 ceph-mon[24793]: purged_snaps scrub ok 2024-03-22T05:17:41.879 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:41 smithi005 ceph-mon[24793]: osd.3 [v2:172.21.15.5:6826/3829723611,v1:172.21.15.5:6827/3829723611] boot 2024-03-22T05:17:41.879 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:41 smithi005 ceph-mon[24793]: osdmap e25: 4 total, 4 up, 4 in 2024-03-22T05:17:41.879 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:41 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T05:17:42.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:41 smithi186 ceph-mon[25720]: purged_snaps scrub starts 2024-03-22T05:17:42.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:41 smithi186 ceph-mon[25720]: purged_snaps scrub ok 2024-03-22T05:17:42.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:41 smithi186 ceph-mon[25720]: osd.3 [v2:172.21.15.5:6826/3829723611,v1:172.21.15.5:6827/3829723611] boot 2024-03-22T05:17:42.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:41 smithi186 ceph-mon[25720]: osdmap e25: 4 total, 4 up, 4 in 2024-03-22T05:17:42.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:41 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T05:17:43.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:42 smithi186 ceph-mon[25720]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-22T05:17:43.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:42 smithi186 ceph-mon[25720]: osdmap e26: 4 total, 4 up, 4 in 2024-03-22T05:17:43.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:42 smithi186 ceph-mon[25720]: Detected new or changed devices on smithi005 2024-03-22T05:17:43.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:42 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:17:43.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:42 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:17:43.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:42 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:17:43.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:42 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:17:43.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:42 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:17:43.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:42 smithi186 ceph-mon[25720]: Adjusting osd_memory_target on smithi005 to 4017M 2024-03-22T05:17:43.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:42 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:17:43.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:42 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:17:43.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:42 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:17:43.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:42 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:17:43.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:42 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:17:43.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:42 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:17:43.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:42 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T05:17:43.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:42 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T05:17:43.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:42 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:17:43.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:42 smithi005 ceph-mon[24793]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-22T05:17:43.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:42 smithi005 ceph-mon[24793]: osdmap e26: 4 total, 4 up, 4 in 2024-03-22T05:17:43.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:42 smithi005 ceph-mon[24793]: Detected new or changed devices on smithi005 2024-03-22T05:17:43.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:42 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:17:43.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:42 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:17:43.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:42 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:17:43.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:42 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:17:43.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:42 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:17:43.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:42 smithi005 ceph-mon[24793]: Adjusting osd_memory_target on smithi005 to 4017M 2024-03-22T05:17:43.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:42 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:17:43.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:42 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:17:43.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:42 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:17:43.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:42 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:17:43.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:42 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:17:43.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:42 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:17:43.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:42 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T05:17:43.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:42 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T05:17:43.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:42 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:17:43.204 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:42 smithi005 ceph-mon[29523]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-22T05:17:43.204 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:42 smithi005 ceph-mon[29523]: osdmap e26: 4 total, 4 up, 4 in 2024-03-22T05:17:43.204 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:42 smithi005 ceph-mon[29523]: Detected new or changed devices on smithi005 2024-03-22T05:17:43.204 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:42 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:17:43.204 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:42 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:17:43.204 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:42 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:17:43.204 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:42 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:17:43.204 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:42 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:17:43.204 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:42 smithi005 ceph-mon[29523]: Adjusting osd_memory_target on smithi005 to 4017M 2024-03-22T05:17:43.204 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:42 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:17:43.204 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:42 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:17:43.204 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:42 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:17:43.204 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:42 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:17:43.204 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:42 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:17:43.205 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:42 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:17:43.205 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:42 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T05:17:43.205 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:42 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T05:17:43.205 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:42 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:17:44.171 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:44 smithi186 ceph-mon[25720]: from='client.34202 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:17:44.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:44 smithi005 ceph-mon[24793]: from='client.34202 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:17:44.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:44 smithi005 ceph-mon[29523]: from='client.34202 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:17:45.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:45 smithi005 ceph-mon[24793]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-22T05:17:45.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:45 smithi005 ceph-mon[24793]: from='client.? 172.21.15.186:0/1321469455' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7313be61-8567-42c4-93a0-8ff3f2e30da3"}]: dispatch 2024-03-22T05:17:45.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:45 smithi005 ceph-mon[24793]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7313be61-8567-42c4-93a0-8ff3f2e30da3"}]: dispatch 2024-03-22T05:17:45.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:45 smithi005 ceph-mon[24793]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7313be61-8567-42c4-93a0-8ff3f2e30da3"}]': finished 2024-03-22T05:17:45.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:45 smithi005 ceph-mon[24793]: osdmap e27: 5 total, 4 up, 5 in 2024-03-22T05:17:45.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:45 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T05:17:45.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:45 smithi005 ceph-mon[29523]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-22T05:17:45.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:45 smithi005 ceph-mon[29523]: from='client.? 172.21.15.186:0/1321469455' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7313be61-8567-42c4-93a0-8ff3f2e30da3"}]: dispatch 2024-03-22T05:17:45.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:45 smithi005 ceph-mon[29523]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7313be61-8567-42c4-93a0-8ff3f2e30da3"}]: dispatch 2024-03-22T05:17:45.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:45 smithi005 ceph-mon[29523]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7313be61-8567-42c4-93a0-8ff3f2e30da3"}]': finished 2024-03-22T05:17:45.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:45 smithi005 ceph-mon[29523]: osdmap e27: 5 total, 4 up, 5 in 2024-03-22T05:17:45.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:45 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T05:17:45.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:45 smithi186 ceph-mon[25720]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-22T05:17:45.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:45 smithi186 ceph-mon[25720]: from='client.? 172.21.15.186:0/1321469455' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7313be61-8567-42c4-93a0-8ff3f2e30da3"}]: dispatch 2024-03-22T05:17:45.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:45 smithi186 ceph-mon[25720]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7313be61-8567-42c4-93a0-8ff3f2e30da3"}]: dispatch 2024-03-22T05:17:45.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:45 smithi186 ceph-mon[25720]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7313be61-8567-42c4-93a0-8ff3f2e30da3"}]': finished 2024-03-22T05:17:45.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:45 smithi186 ceph-mon[25720]: osdmap e27: 5 total, 4 up, 5 in 2024-03-22T05:17:45.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:45 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T05:17:46.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:46 smithi005 ceph-mon[24793]: from='client.? 172.21.15.186:0/67911752' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T05:17:46.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:46 smithi005 ceph-mon[29523]: from='client.? 172.21.15.186:0/67911752' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T05:17:46.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:46 smithi186 ceph-mon[25720]: from='client.? 172.21.15.186:0/67911752' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T05:17:47.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:47 smithi005 ceph-mon[24793]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-22T05:17:47.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:47 smithi005 ceph-mon[29523]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-22T05:17:47.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:47 smithi186 ceph-mon[25720]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-22T05:17:49.222 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:49 smithi186 ceph-mon[25720]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-22T05:17:49.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:49 smithi005 ceph-mon[24793]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-22T05:17:49.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:49 smithi005 ceph-mon[29523]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-22T05:17:51.432 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:51 smithi186 ceph-mon[25720]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-22T05:17:51.432 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:51 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-22T05:17:51.432 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:51 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:17:51.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:51 smithi005 ceph-mon[24793]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-22T05:17:51.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:51 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-22T05:17:51.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:51 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:17:51.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:51 smithi005 ceph-mon[29523]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-22T05:17:51.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:51 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-22T05:17:51.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:51 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:17:52.363 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:52 smithi186 ceph-mon[25720]: Deploying daemon osd.4 on smithi186 2024-03-22T05:17:52.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:52 smithi005 ceph-mon[24793]: Deploying daemon osd.4 on smithi186 2024-03-22T05:17:52.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:52 smithi005 ceph-mon[29523]: Deploying daemon osd.4 on smithi186 2024-03-22T05:17:53.413 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:53 smithi186 ceph-mon[25720]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-22T05:17:53.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:53 smithi005 ceph-mon[24793]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-22T05:17:53.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:53 smithi005 ceph-mon[29523]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-22T05:17:55.430 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:55 smithi186 ceph-mon[25720]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-22T05:17:55.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:55 smithi005 ceph-mon[24793]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-22T05:17:55.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:55 smithi005 ceph-mon[29523]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-22T05:17:56.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:56 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:17:56.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:56 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:17:56.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:56 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:17:56.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:56 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:17:56.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:56 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:17:56.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:56 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:17:56.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:56 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:17:56.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:56 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:17:56.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:56 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:17:56.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:56 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:17:56.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:56 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:17:56.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:56 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:17:57.220 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:57 smithi186 ceph-mon[25720]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-22T05:17:57.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:57 smithi005 ceph-mon[24793]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-22T05:17:57.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:57 smithi005 ceph-mon[29523]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-22T05:17:58.359 INFO:teuthology.orchestra.run.smithi186.stdout:Created osd(s) 4 on host 'smithi186' 2024-03-22T05:17:58.953 DEBUG:teuthology.orchestra.run.smithi186:osd.4> sudo journalctl -f -n 0 -u ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@osd.4.service 2024-03-22T05:17:58.996 INFO:tasks.cephadm:Deploying osd.5 on smithi186 with /dev/vg_nvme/lv_3... 2024-03-22T05:17:58.996 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-22T05:17:59.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:59 smithi186 ceph-mon[25720]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-22T05:17:59.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:59 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:17:59.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:59 smithi186 ceph-mon[25720]: from='osd.4 [v2:172.21.15.186:6800/2861669931,v1:172.21.15.186:6801/2861669931]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-22T05:17:59.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:17:59 smithi186 ceph-mon[25720]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-22T05:17:59.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:59 smithi005 ceph-mon[24793]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-22T05:17:59.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:59 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:17:59.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:59 smithi005 ceph-mon[24793]: from='osd.4 [v2:172.21.15.186:6800/2861669931,v1:172.21.15.186:6801/2861669931]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-22T05:17:59.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:17:59 smithi005 ceph-mon[24793]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-22T05:17:59.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:59 smithi005 ceph-mon[29523]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-22T05:17:59.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:59 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:17:59.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:59 smithi005 ceph-mon[29523]: from='osd.4 [v2:172.21.15.186:6800/2861669931,v1:172.21.15.186:6801/2861669931]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-22T05:17:59.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:17:59 smithi005 ceph-mon[29523]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-22T05:18:00.645 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 22 05:18:00 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-4[29127]: 2024-03-22T05:18:00.365+0000 7f10adee9700 -1 osd.4 0 waiting for initial osdmap 2024-03-22T05:18:00.645 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 22 05:18:00 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-4[29127]: 2024-03-22T05:18:00.369+0000 7f10a9882700 -1 osd.4 29 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-22T05:18:00.645 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:00 smithi186 ceph-mon[25720]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-22T05:18:00.645 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:00 smithi186 ceph-mon[25720]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-22T05:18:00.645 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:00 smithi186 ceph-mon[25720]: osdmap e28: 5 total, 4 up, 5 in 2024-03-22T05:18:00.645 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:00 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T05:18:00.646 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:00 smithi186 ceph-mon[25720]: from='osd.4 [v2:172.21.15.186:6800/2861669931,v1:172.21.15.186:6801/2861669931]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-03-22T05:18:00.646 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:00 smithi186 ceph-mon[25720]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-03-22T05:18:00.646 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:00 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:18:00.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:00 smithi005 ceph-mon[24793]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-22T05:18:00.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:00 smithi005 ceph-mon[24793]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-22T05:18:00.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:00 smithi005 ceph-mon[24793]: osdmap e28: 5 total, 4 up, 5 in 2024-03-22T05:18:00.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:00 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T05:18:00.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:00 smithi005 ceph-mon[24793]: from='osd.4 [v2:172.21.15.186:6800/2861669931,v1:172.21.15.186:6801/2861669931]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-03-22T05:18:00.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:00 smithi005 ceph-mon[24793]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-03-22T05:18:00.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:00 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:18:00.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:00 smithi005 ceph-mon[29523]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-22T05:18:00.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:00 smithi005 ceph-mon[29523]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-22T05:18:00.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:00 smithi005 ceph-mon[29523]: osdmap e28: 5 total, 4 up, 5 in 2024-03-22T05:18:00.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:00 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T05:18:00.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:00 smithi005 ceph-mon[29523]: from='osd.4 [v2:172.21.15.186:6800/2861669931,v1:172.21.15.186:6801/2861669931]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-03-22T05:18:00.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:00 smithi005 ceph-mon[29523]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-03-22T05:18:00.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:00 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:18:01.330 INFO:teuthology.orchestra.run.smithi186.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-22T05:18:01.330 INFO:teuthology.orchestra.run.smithi186.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-22T05:18:01.330 INFO:teuthology.orchestra.run.smithi186.stderr: stderr: 10+0 records in 2024-03-22T05:18:01.330 INFO:teuthology.orchestra.run.smithi186.stderr:10+0 records out 2024-03-22T05:18:01.330 INFO:teuthology.orchestra.run.smithi186.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0127423 s, 823 MB/s 2024-03-22T05:18:01.330 INFO:teuthology.orchestra.run.smithi186.stderr:--> Zapping successful for: 2024-03-22T05:18:01.582 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:01 smithi186 ceph-mon[25720]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-03-22T05:18:01.582 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:01 smithi186 ceph-mon[25720]: osdmap e29: 5 total, 4 up, 5 in 2024-03-22T05:18:01.582 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:01 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T05:18:01.582 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:01 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T05:18:01.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:01 smithi005 ceph-mon[24793]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-03-22T05:18:01.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:01 smithi005 ceph-mon[24793]: osdmap e29: 5 total, 4 up, 5 in 2024-03-22T05:18:01.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:01 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T05:18:01.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:01 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T05:18:01.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:01 smithi005 ceph-mon[29523]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-03-22T05:18:01.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:01 smithi005 ceph-mon[29523]: osdmap e29: 5 total, 4 up, 5 in 2024-03-22T05:18:01.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:01 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T05:18:01.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:01 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T05:18:02.310 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi186:vg_nvme/lv_3 2024-03-22T05:18:02.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:02 smithi186 ceph-mon[25720]: purged_snaps scrub starts 2024-03-22T05:18:02.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:02 smithi186 ceph-mon[25720]: purged_snaps scrub ok 2024-03-22T05:18:02.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:02 smithi186 ceph-mon[25720]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-22T05:18:02.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:02 smithi186 ceph-mon[25720]: osd.4 [v2:172.21.15.186:6800/2861669931,v1:172.21.15.186:6801/2861669931] boot 2024-03-22T05:18:02.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:02 smithi186 ceph-mon[25720]: osdmap e30: 5 total, 5 up, 5 in 2024-03-22T05:18:02.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:02 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T05:18:02.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:02 smithi005 ceph-mon[24793]: purged_snaps scrub starts 2024-03-22T05:18:02.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:02 smithi005 ceph-mon[24793]: purged_snaps scrub ok 2024-03-22T05:18:02.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:02 smithi005 ceph-mon[24793]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-22T05:18:02.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:02 smithi005 ceph-mon[24793]: osd.4 [v2:172.21.15.186:6800/2861669931,v1:172.21.15.186:6801/2861669931] boot 2024-03-22T05:18:02.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:02 smithi005 ceph-mon[24793]: osdmap e30: 5 total, 5 up, 5 in 2024-03-22T05:18:02.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:02 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T05:18:02.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:02 smithi005 ceph-mon[29523]: purged_snaps scrub starts 2024-03-22T05:18:02.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:02 smithi005 ceph-mon[29523]: purged_snaps scrub ok 2024-03-22T05:18:02.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:02 smithi005 ceph-mon[29523]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-22T05:18:02.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:02 smithi005 ceph-mon[29523]: osd.4 [v2:172.21.15.186:6800/2861669931,v1:172.21.15.186:6801/2861669931] boot 2024-03-22T05:18:02.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:02 smithi005 ceph-mon[29523]: osdmap e30: 5 total, 5 up, 5 in 2024-03-22T05:18:02.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:02 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T05:18:03.537 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:03 smithi186 ceph-mon[25720]: osdmap e31: 5 total, 5 up, 5 in 2024-03-22T05:18:03.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:03 smithi005 ceph-mon[24793]: osdmap e31: 5 total, 5 up, 5 in 2024-03-22T05:18:03.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:03 smithi005 ceph-mon[29523]: osdmap e31: 5 total, 5 up, 5 in 2024-03-22T05:18:04.474 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:04 smithi186 ceph-mon[25720]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-03-22T05:18:04.474 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:04 smithi186 ceph-mon[25720]: osdmap e32: 5 total, 5 up, 5 in 2024-03-22T05:18:04.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:04 smithi005 ceph-mon[24793]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-03-22T05:18:04.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:04 smithi005 ceph-mon[24793]: osdmap e32: 5 total, 5 up, 5 in 2024-03-22T05:18:04.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:04 smithi005 ceph-mon[29523]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-03-22T05:18:04.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:04 smithi005 ceph-mon[29523]: osdmap e32: 5 total, 5 up, 5 in 2024-03-22T05:18:05.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:05 smithi005 ceph-mon[24793]: osdmap e33: 5 total, 5 up, 5 in 2024-03-22T05:18:05.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:05 smithi005 ceph-mon[24793]: from='client.34229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:18:05.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:05 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T05:18:05.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:05 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T05:18:05.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:05 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:18:05.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:05 smithi005 ceph-mon[24793]: Detected new or changed devices on smithi186 2024-03-22T05:18:05.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:05 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:18:05.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:05 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:18:05.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:05 smithi005 ceph-mon[24793]: Adjusting osd_memory_target on smithi186 to 17088M 2024-03-22T05:18:05.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:05 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:18:05.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:05 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:18:05.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:05 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:18:05.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:05 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:18:05.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:05 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:18:05.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:05 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:18:05.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:05 smithi005 ceph-mon[29523]: osdmap e33: 5 total, 5 up, 5 in 2024-03-22T05:18:05.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:05 smithi005 ceph-mon[29523]: from='client.34229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:18:05.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:05 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T05:18:05.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:05 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T05:18:05.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:05 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:18:05.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:05 smithi005 ceph-mon[29523]: Detected new or changed devices on smithi186 2024-03-22T05:18:05.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:05 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:18:05.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:05 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:18:05.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:05 smithi005 ceph-mon[29523]: Adjusting osd_memory_target on smithi186 to 17088M 2024-03-22T05:18:05.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:05 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:18:05.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:05 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:18:05.705 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:05 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:18:05.705 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:05 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:18:05.705 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:05 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:18:05.705 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:05 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:18:05.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:05 smithi186 ceph-mon[25720]: osdmap e33: 5 total, 5 up, 5 in 2024-03-22T05:18:05.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:05 smithi186 ceph-mon[25720]: from='client.34229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:18:05.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:05 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T05:18:05.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:05 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T05:18:05.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:05 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:18:05.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:05 smithi186 ceph-mon[25720]: Detected new or changed devices on smithi186 2024-03-22T05:18:05.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:05 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:18:05.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:05 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:18:05.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:05 smithi186 ceph-mon[25720]: Adjusting osd_memory_target on smithi186 to 17088M 2024-03-22T05:18:05.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:05 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:18:05.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:05 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:18:05.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:05 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:18:05.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:05 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:18:05.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:05 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:18:05.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:05 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:18:06.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:06 smithi005 ceph-mon[24793]: pgmap v111: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-03-22T05:18:06.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:06 smithi005 ceph-mon[24793]: osdmap e34: 5 total, 5 up, 5 in 2024-03-22T05:18:06.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:06 smithi005 ceph-mon[29523]: pgmap v111: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-03-22T05:18:06.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:06 smithi005 ceph-mon[29523]: osdmap e34: 5 total, 5 up, 5 in 2024-03-22T05:18:06.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:06 smithi186 ceph-mon[25720]: pgmap v111: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-03-22T05:18:06.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:06 smithi186 ceph-mon[25720]: osdmap e34: 5 total, 5 up, 5 in 2024-03-22T05:18:07.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:07 smithi005 ceph-mon[24793]: from='client.? 172.21.15.186:0/335044028' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "edeed5ee-7262-4ce5-a32f-cef71b2d6acf"}]: dispatch 2024-03-22T05:18:07.721 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:07 smithi005 ceph-mon[24793]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "edeed5ee-7262-4ce5-a32f-cef71b2d6acf"}]: dispatch 2024-03-22T05:18:07.721 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:07 smithi005 ceph-mon[24793]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "edeed5ee-7262-4ce5-a32f-cef71b2d6acf"}]': finished 2024-03-22T05:18:07.721 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:07 smithi005 ceph-mon[24793]: osdmap e35: 6 total, 5 up, 6 in 2024-03-22T05:18:07.721 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:07 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T05:18:07.721 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:07 smithi005 ceph-mon[24793]: from='client.? 172.21.15.186:0/4232569613' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T05:18:07.722 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:07 smithi005 ceph-mon[29523]: from='client.? 172.21.15.186:0/335044028' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "edeed5ee-7262-4ce5-a32f-cef71b2d6acf"}]: dispatch 2024-03-22T05:18:07.722 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:07 smithi005 ceph-mon[29523]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "edeed5ee-7262-4ce5-a32f-cef71b2d6acf"}]: dispatch 2024-03-22T05:18:07.722 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:07 smithi005 ceph-mon[29523]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "edeed5ee-7262-4ce5-a32f-cef71b2d6acf"}]': finished 2024-03-22T05:18:07.722 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:07 smithi005 ceph-mon[29523]: osdmap e35: 6 total, 5 up, 6 in 2024-03-22T05:18:07.722 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:07 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T05:18:07.722 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:07 smithi005 ceph-mon[29523]: from='client.? 172.21.15.186:0/4232569613' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T05:18:07.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:07 smithi186 ceph-mon[25720]: from='client.? 172.21.15.186:0/335044028' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "edeed5ee-7262-4ce5-a32f-cef71b2d6acf"}]: dispatch 2024-03-22T05:18:07.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:07 smithi186 ceph-mon[25720]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "edeed5ee-7262-4ce5-a32f-cef71b2d6acf"}]: dispatch 2024-03-22T05:18:07.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:07 smithi186 ceph-mon[25720]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "edeed5ee-7262-4ce5-a32f-cef71b2d6acf"}]': finished 2024-03-22T05:18:07.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:07 smithi186 ceph-mon[25720]: osdmap e35: 6 total, 5 up, 6 in 2024-03-22T05:18:07.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:07 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T05:18:07.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:07 smithi186 ceph-mon[25720]: from='client.? 172.21.15.186:0/4232569613' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T05:18:09.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:09 smithi005 ceph-mon[24793]: pgmap v114: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-03-22T05:18:09.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:09 smithi005 ceph-mon[29523]: pgmap v114: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-03-22T05:18:09.453 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:18:09 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:18:09.160+0000 7f13a899d700 -1 calc_pg_upmaps abort due to max <= 0 2024-03-22T05:18:09.526 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:09 smithi186 ceph-mon[25720]: pgmap v114: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-03-22T05:18:10.134 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:10 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-22T05:18:10.134 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:10 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-22T05:18:10.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:10 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-22T05:18:10.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:10 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-22T05:18:10.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:10 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-22T05:18:10.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:10 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-22T05:18:11.339 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:11 smithi186 ceph-mon[25720]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-03-22T05:18:11.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:11 smithi005 ceph-mon[24793]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-03-22T05:18:11.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:11 smithi005 ceph-mon[29523]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-03-22T05:18:13.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:13 smithi186 ceph-mon[25720]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-22T05:18:13.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:13 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-22T05:18:13.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:13 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:18:13.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:13 smithi005 ceph-mon[24793]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-22T05:18:13.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:13 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-22T05:18:13.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:13 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:18:13.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:13 smithi005 ceph-mon[29523]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-22T05:18:13.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:13 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-22T05:18:13.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:13 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:18:14.286 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:14 smithi186 ceph-mon[25720]: Deploying daemon osd.5 on smithi186 2024-03-22T05:18:14.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:14 smithi005 ceph-mon[24793]: Deploying daemon osd.5 on smithi186 2024-03-22T05:18:14.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:14 smithi005 ceph-mon[29523]: Deploying daemon osd.5 on smithi186 2024-03-22T05:18:15.368 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:15 smithi186 ceph-mon[25720]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-22T05:18:15.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:15 smithi005 ceph-mon[24793]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-22T05:18:15.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:15 smithi005 ceph-mon[29523]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-22T05:18:17.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:17 smithi186 ceph-mon[25720]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-22T05:18:17.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:17 smithi005 ceph-mon[24793]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-22T05:18:17.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:17 smithi005 ceph-mon[29523]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-22T05:18:19.169 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:18 smithi186 ceph-mon[25720]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-22T05:18:19.169 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:18 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:18:19.169 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:18 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:18:19.169 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:18 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:18:19.170 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:18 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:18:19.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:18 smithi005 ceph-mon[24793]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-22T05:18:19.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:18 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:18:19.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:18 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:18:19.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:18 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:18:19.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:18 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:18:19.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:18 smithi005 ceph-mon[29523]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-22T05:18:19.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:18 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:18:19.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:18 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:18:19.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:18 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:18:19.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:18 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:18:20.855 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:20 smithi186 ceph-mon[25720]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-22T05:18:20.885 INFO:teuthology.orchestra.run.smithi186.stdout:Created osd(s) 5 on host 'smithi186' 2024-03-22T05:18:21.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:20 smithi005 ceph-mon[24793]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-22T05:18:21.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:20 smithi005 ceph-mon[29523]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-22T05:18:21.505 DEBUG:teuthology.orchestra.run.smithi186:osd.5> sudo journalctl -f -n 0 -u ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@osd.5.service 2024-03-22T05:18:21.507 INFO:tasks.cephadm:Deploying osd.6 on smithi186 with /dev/vg_nvme/lv_2... 2024-03-22T05:18:21.507 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-22T05:18:22.170 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:21 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:18:22.170 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:21 smithi186 ceph-mon[25720]: from='osd.5 [v2:172.21.15.186:6808/554926692,v1:172.21.15.186:6809/554926692]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-22T05:18:22.170 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:21 smithi186 ceph-mon[25720]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-22T05:18:22.170 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:21 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:18:22.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:21 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:18:22.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:21 smithi005 ceph-mon[24793]: from='osd.5 [v2:172.21.15.186:6808/554926692,v1:172.21.15.186:6809/554926692]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-22T05:18:22.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:21 smithi005 ceph-mon[24793]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-22T05:18:22.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:21 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:18:22.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:21 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:18:22.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:21 smithi005 ceph-mon[29523]: from='osd.5 [v2:172.21.15.186:6808/554926692,v1:172.21.15.186:6809/554926692]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-22T05:18:22.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:21 smithi005 ceph-mon[29523]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-22T05:18:22.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:21 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:18:22.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:22 smithi186 ceph-mon[25720]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-22T05:18:22.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:22 smithi186 ceph-mon[25720]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-22T05:18:22.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:22 smithi186 ceph-mon[25720]: osdmap e36: 6 total, 5 up, 6 in 2024-03-22T05:18:22.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:22 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T05:18:22.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:22 smithi186 ceph-mon[25720]: from='osd.5 [v2:172.21.15.186:6808/554926692,v1:172.21.15.186:6809/554926692]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-03-22T05:18:22.962 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:22 smithi186 ceph-mon[25720]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-03-22T05:18:22.962 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:18:22 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-5[32003]: 2024-03-22T05:18:22.894+0000 7f56221f0700 -1 osd.5 0 waiting for initial osdmap 2024-03-22T05:18:22.963 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:18:22 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-5[32003]: 2024-03-22T05:18:22.899+0000 7f561cb87700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-22T05:18:23.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:22 smithi005 ceph-mon[24793]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-22T05:18:23.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:22 smithi005 ceph-mon[24793]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-22T05:18:23.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:22 smithi005 ceph-mon[24793]: osdmap e36: 6 total, 5 up, 6 in 2024-03-22T05:18:23.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:22 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T05:18:23.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:22 smithi005 ceph-mon[24793]: from='osd.5 [v2:172.21.15.186:6808/554926692,v1:172.21.15.186:6809/554926692]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-03-22T05:18:23.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:22 smithi005 ceph-mon[24793]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-03-22T05:18:23.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:22 smithi005 ceph-mon[29523]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-22T05:18:23.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:22 smithi005 ceph-mon[29523]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-22T05:18:23.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:22 smithi005 ceph-mon[29523]: osdmap e36: 6 total, 5 up, 6 in 2024-03-22T05:18:23.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:22 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T05:18:23.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:22 smithi005 ceph-mon[29523]: from='osd.5 [v2:172.21.15.186:6808/554926692,v1:172.21.15.186:6809/554926692]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-03-22T05:18:23.204 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:22 smithi005 ceph-mon[29523]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-03-22T05:18:23.471 INFO:teuthology.orchestra.run.smithi186.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-22T05:18:23.471 INFO:teuthology.orchestra.run.smithi186.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-22T05:18:23.471 INFO:teuthology.orchestra.run.smithi186.stderr: stderr: 10+0 records in 2024-03-22T05:18:23.471 INFO:teuthology.orchestra.run.smithi186.stderr:10+0 records out 2024-03-22T05:18:23.471 INFO:teuthology.orchestra.run.smithi186.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0126947 s, 826 MB/s 2024-03-22T05:18:23.471 INFO:teuthology.orchestra.run.smithi186.stderr:--> Zapping successful for: 2024-03-22T05:18:24.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:23 smithi005 ceph-mon[24793]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-03-22T05:18:24.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:23 smithi005 ceph-mon[24793]: osdmap e37: 6 total, 5 up, 6 in 2024-03-22T05:18:24.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:23 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T05:18:24.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:23 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T05:18:24.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:23 smithi005 ceph-mon[24793]: osd.5 [v2:172.21.15.186:6808/554926692,v1:172.21.15.186:6809/554926692] boot 2024-03-22T05:18:24.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:23 smithi005 ceph-mon[24793]: osdmap e38: 6 total, 6 up, 6 in 2024-03-22T05:18:24.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:23 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T05:18:24.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:23 smithi005 ceph-mon[29523]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-03-22T05:18:24.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:23 smithi005 ceph-mon[29523]: osdmap e37: 6 total, 5 up, 6 in 2024-03-22T05:18:24.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:23 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T05:18:24.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:23 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T05:18:24.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:23 smithi005 ceph-mon[29523]: osd.5 [v2:172.21.15.186:6808/554926692,v1:172.21.15.186:6809/554926692] boot 2024-03-22T05:18:24.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:23 smithi005 ceph-mon[29523]: osdmap e38: 6 total, 6 up, 6 in 2024-03-22T05:18:24.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:23 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T05:18:24.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:23 smithi186 ceph-mon[25720]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-03-22T05:18:24.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:23 smithi186 ceph-mon[25720]: osdmap e37: 6 total, 5 up, 6 in 2024-03-22T05:18:24.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:23 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T05:18:24.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:23 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T05:18:24.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:23 smithi186 ceph-mon[25720]: osd.5 [v2:172.21.15.186:6808/554926692,v1:172.21.15.186:6809/554926692] boot 2024-03-22T05:18:24.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:23 smithi186 ceph-mon[25720]: osdmap e38: 6 total, 6 up, 6 in 2024-03-22T05:18:24.210 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:23 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T05:18:24.730 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi186:vg_nvme/lv_2 2024-03-22T05:18:25.034 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:24 smithi186 ceph-mon[25720]: purged_snaps scrub starts 2024-03-22T05:18:25.034 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:24 smithi186 ceph-mon[25720]: purged_snaps scrub ok 2024-03-22T05:18:25.034 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:24 smithi186 ceph-mon[25720]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-22T05:18:25.034 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:24 smithi186 ceph-mon[25720]: osdmap e39: 6 total, 6 up, 6 in 2024-03-22T05:18:25.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:24 smithi005 ceph-mon[24793]: purged_snaps scrub starts 2024-03-22T05:18:25.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:24 smithi005 ceph-mon[24793]: purged_snaps scrub ok 2024-03-22T05:18:25.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:24 smithi005 ceph-mon[24793]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-22T05:18:25.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:24 smithi005 ceph-mon[24793]: osdmap e39: 6 total, 6 up, 6 in 2024-03-22T05:18:25.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:24 smithi005 ceph-mon[29523]: purged_snaps scrub starts 2024-03-22T05:18:25.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:24 smithi005 ceph-mon[29523]: purged_snaps scrub ok 2024-03-22T05:18:25.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:24 smithi005 ceph-mon[29523]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-22T05:18:25.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:24 smithi005 ceph-mon[29523]: osdmap e39: 6 total, 6 up, 6 in 2024-03-22T05:18:26.189 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:26 smithi186 ceph-mon[25720]: osdmap e40: 6 total, 6 up, 6 in 2024-03-22T05:18:26.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:26 smithi005 ceph-mon[24793]: osdmap e40: 6 total, 6 up, 6 in 2024-03-22T05:18:26.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:26 smithi005 ceph-mon[29523]: osdmap e40: 6 total, 6 up, 6 in 2024-03-22T05:18:27.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:27 smithi186 ceph-mon[25720]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-03-22T05:18:27.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:27 smithi186 ceph-mon[25720]: osdmap e41: 6 total, 6 up, 6 in 2024-03-22T05:18:27.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:27 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T05:18:27.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:27 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T05:18:27.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:27 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:18:27.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:27 smithi005 ceph-mon[24793]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-03-22T05:18:27.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:27 smithi005 ceph-mon[24793]: osdmap e41: 6 total, 6 up, 6 in 2024-03-22T05:18:27.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:27 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T05:18:27.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:27 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T05:18:27.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:27 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:18:27.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:27 smithi005 ceph-mon[29523]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-03-22T05:18:27.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:27 smithi005 ceph-mon[29523]: osdmap e41: 6 total, 6 up, 6 in 2024-03-22T05:18:27.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:27 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T05:18:27.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:27 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T05:18:27.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:27 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:18:28.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:28 smithi005 ceph-mon[29523]: from='client.34256 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:18:28.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:28 smithi005 ceph-mon[29523]: osdmap e42: 6 total, 6 up, 6 in 2024-03-22T05:18:28.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:28 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:18:28.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:28 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:18:28.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:28 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:18:28.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:28 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:18:28.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:28 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:18:28.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:28 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:18:28.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:28 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:18:28.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:28 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:18:28.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:28 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:18:28.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:28 smithi005 ceph-mon[24793]: from='client.34256 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:18:28.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:28 smithi005 ceph-mon[24793]: osdmap e42: 6 total, 6 up, 6 in 2024-03-22T05:18:28.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:28 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:18:28.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:28 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:18:28.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:28 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:18:28.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:28 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:18:28.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:28 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:18:28.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:28 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:18:28.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:28 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:18:28.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:28 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:18:28.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:28 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:18:28.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:28 smithi186 ceph-mon[25720]: from='client.34256 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:18:28.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:28 smithi186 ceph-mon[25720]: osdmap e42: 6 total, 6 up, 6 in 2024-03-22T05:18:28.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:28 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:18:28.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:28 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:18:28.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:28 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:18:28.478 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:28 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:18:28.478 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:28 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:18:28.478 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:28 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:18:28.478 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:28 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:18:28.478 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:28 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:18:28.478 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:28 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:18:29.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:29 smithi186 ceph-mon[25720]: pgmap v131: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-03-22T05:18:29.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:29 smithi186 ceph-mon[25720]: Detected new or changed devices on smithi186 2024-03-22T05:18:29.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:29 smithi186 ceph-mon[25720]: Adjusting osd_memory_target on smithi186 to 8544M 2024-03-22T05:18:29.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:29 smithi005 ceph-mon[24793]: pgmap v131: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-03-22T05:18:29.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:29 smithi005 ceph-mon[24793]: Detected new or changed devices on smithi186 2024-03-22T05:18:29.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:29 smithi005 ceph-mon[24793]: Adjusting osd_memory_target on smithi186 to 8544M 2024-03-22T05:18:29.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:29 smithi005 ceph-mon[29523]: pgmap v131: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-03-22T05:18:29.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:29 smithi005 ceph-mon[29523]: Detected new or changed devices on smithi186 2024-03-22T05:18:29.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:29 smithi005 ceph-mon[29523]: Adjusting osd_memory_target on smithi186 to 8544M 2024-03-22T05:18:30.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:30 smithi005 ceph-mon[24793]: from='client.? 172.21.15.186:0/904832441' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b979e26c-6b94-434f-8c95-9b9a8713eb78"}]: dispatch 2024-03-22T05:18:30.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:30 smithi005 ceph-mon[24793]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b979e26c-6b94-434f-8c95-9b9a8713eb78"}]: dispatch 2024-03-22T05:18:30.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:30 smithi005 ceph-mon[24793]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b979e26c-6b94-434f-8c95-9b9a8713eb78"}]': finished 2024-03-22T05:18:30.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:30 smithi005 ceph-mon[24793]: osdmap e43: 7 total, 6 up, 7 in 2024-03-22T05:18:30.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:30 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-22T05:18:30.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:30 smithi005 ceph-mon[29523]: from='client.? 172.21.15.186:0/904832441' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b979e26c-6b94-434f-8c95-9b9a8713eb78"}]: dispatch 2024-03-22T05:18:30.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:30 smithi005 ceph-mon[29523]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b979e26c-6b94-434f-8c95-9b9a8713eb78"}]: dispatch 2024-03-22T05:18:30.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:30 smithi005 ceph-mon[29523]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b979e26c-6b94-434f-8c95-9b9a8713eb78"}]': finished 2024-03-22T05:18:30.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:30 smithi005 ceph-mon[29523]: osdmap e43: 7 total, 6 up, 7 in 2024-03-22T05:18:30.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:30 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-22T05:18:30.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:30 smithi186 ceph-mon[25720]: from='client.? 172.21.15.186:0/904832441' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b979e26c-6b94-434f-8c95-9b9a8713eb78"}]: dispatch 2024-03-22T05:18:30.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:30 smithi186 ceph-mon[25720]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b979e26c-6b94-434f-8c95-9b9a8713eb78"}]: dispatch 2024-03-22T05:18:30.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:30 smithi186 ceph-mon[25720]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b979e26c-6b94-434f-8c95-9b9a8713eb78"}]': finished 2024-03-22T05:18:30.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:30 smithi186 ceph-mon[25720]: osdmap e43: 7 total, 6 up, 7 in 2024-03-22T05:18:30.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:30 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-22T05:18:31.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:31 smithi186 ceph-mon[25720]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-22T05:18:31.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:31 smithi186 ceph-mon[25720]: from='client.? 172.21.15.186:0/1318251316' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T05:18:31.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:31 smithi005 ceph-mon[24793]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-22T05:18:31.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:31 smithi005 ceph-mon[24793]: from='client.? 172.21.15.186:0/1318251316' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T05:18:31.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:31 smithi005 ceph-mon[29523]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-22T05:18:31.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:31 smithi005 ceph-mon[29523]: from='client.? 172.21.15.186:0/1318251316' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T05:18:32.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:32 smithi005 ceph-mon[29523]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-22T05:18:32.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:32 smithi005 ceph-mon[24793]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-22T05:18:32.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:32 smithi186 ceph-mon[25720]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-22T05:18:35.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:35 smithi005 ceph-mon[29523]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-22T05:18:35.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:35 smithi005 ceph-mon[24793]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-22T05:18:35.490 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:35 smithi186 ceph-mon[25720]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-22T05:18:36.406 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:36 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-22T05:18:36.407 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:36 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:18:36.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:36 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-22T05:18:36.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:36 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:18:36.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:36 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-22T05:18:36.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:36 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:18:37.365 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:37 smithi186 ceph-mon[25720]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-22T05:18:37.365 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:37 smithi186 ceph-mon[25720]: Deploying daemon osd.6 on smithi186 2024-03-22T05:18:37.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:37 smithi005 ceph-mon[24793]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-22T05:18:37.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:37 smithi005 ceph-mon[24793]: Deploying daemon osd.6 on smithi186 2024-03-22T05:18:37.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:37 smithi005 ceph-mon[29523]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-22T05:18:37.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:37 smithi005 ceph-mon[29523]: Deploying daemon osd.6 on smithi186 2024-03-22T05:18:39.291 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:39 smithi186 ceph-mon[25720]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-22T05:18:39.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:39 smithi005 ceph-mon[24793]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-22T05:18:39.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:39 smithi005 ceph-mon[29523]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-22T05:18:41.259 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:41 smithi005 ceph-mon[24793]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-22T05:18:41.259 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:41 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:18:41.259 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:41 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:18:41.260 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:41 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:18:41.260 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:41 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:18:41.260 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:41 smithi005 ceph-mon[29523]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-22T05:18:41.260 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:41 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:18:41.260 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:41 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:18:41.260 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:41 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:18:41.260 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:41 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:18:41.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:41 smithi186 ceph-mon[25720]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-22T05:18:41.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:41 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:18:41.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:41 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:18:41.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:41 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:18:41.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:41 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:18:43.371 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:43 smithi186 ceph-mon[25720]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-22T05:18:43.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:43 smithi005 ceph-mon[24793]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-22T05:18:43.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:43 smithi005 ceph-mon[29523]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-22T05:18:44.422 INFO:teuthology.orchestra.run.smithi186.stdout:Created osd(s) 6 on host 'smithi186' 2024-03-22T05:18:44.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:44 smithi186 ceph-mon[25720]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-22T05:18:44.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:44 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:18:44.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:44 smithi186 ceph-mon[25720]: from='osd.6 [v2:172.21.15.186:6816/2574435174,v1:172.21.15.186:6817/2574435174]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-22T05:18:44.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:44 smithi186 ceph-mon[25720]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-22T05:18:44.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:44 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:18:44.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:44 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:18:44.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:44 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:18:44.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:44 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:18:44.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:44 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:18:44.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:44 smithi005 ceph-mon[24793]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-22T05:18:44.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:44 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:18:44.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:44 smithi005 ceph-mon[24793]: from='osd.6 [v2:172.21.15.186:6816/2574435174,v1:172.21.15.186:6817/2574435174]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-22T05:18:44.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:44 smithi005 ceph-mon[24793]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-22T05:18:44.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:44 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:18:44.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:44 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:18:44.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:44 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:18:44.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:44 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:18:44.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:44 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:18:44.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:44 smithi005 ceph-mon[29523]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-22T05:18:44.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:44 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:18:44.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:44 smithi005 ceph-mon[29523]: from='osd.6 [v2:172.21.15.186:6816/2574435174,v1:172.21.15.186:6817/2574435174]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-22T05:18:44.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:44 smithi005 ceph-mon[29523]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-22T05:18:44.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:44 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:18:44.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:44 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:18:44.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:44 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:18:44.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:44 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:18:44.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:44 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:18:45.106 DEBUG:teuthology.orchestra.run.smithi186:osd.6> sudo journalctl -f -n 0 -u ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@osd.6.service 2024-03-22T05:18:45.109 INFO:tasks.cephadm:Deploying osd.7 on smithi186 with /dev/vg_nvme/lv_1... 2024-03-22T05:18:45.109 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-22T05:18:45.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:45 smithi186 ceph-mon[25720]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-22T05:18:45.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:45 smithi186 ceph-mon[25720]: osdmap e44: 7 total, 6 up, 7 in 2024-03-22T05:18:45.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:45 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-22T05:18:45.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:45 smithi186 ceph-mon[25720]: from='osd.6 [v2:172.21.15.186:6816/2574435174,v1:172.21.15.186:6817/2574435174]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-03-22T05:18:45.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:45 smithi186 ceph-mon[25720]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-03-22T05:18:45.795 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:18:45 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6[34894]: 2024-03-22T05:18:45.567+0000 7f40a5e77700 -1 osd.6 0 waiting for initial osdmap 2024-03-22T05:18:45.795 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:18:45 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6[34894]: 2024-03-22T05:18:45.575+0000 7f409f00b700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-22T05:18:45.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:45 smithi005 ceph-mon[24793]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-22T05:18:45.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:45 smithi005 ceph-mon[24793]: osdmap e44: 7 total, 6 up, 7 in 2024-03-22T05:18:45.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:45 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-22T05:18:45.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:45 smithi005 ceph-mon[24793]: from='osd.6 [v2:172.21.15.186:6816/2574435174,v1:172.21.15.186:6817/2574435174]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-03-22T05:18:45.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:45 smithi005 ceph-mon[24793]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-03-22T05:18:45.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:45 smithi005 ceph-mon[29523]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-22T05:18:45.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:45 smithi005 ceph-mon[29523]: osdmap e44: 7 total, 6 up, 7 in 2024-03-22T05:18:45.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:45 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-22T05:18:45.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:45 smithi005 ceph-mon[29523]: from='osd.6 [v2:172.21.15.186:6816/2574435174,v1:172.21.15.186:6817/2574435174]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-03-22T05:18:45.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:45 smithi005 ceph-mon[29523]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-03-22T05:18:46.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:46 smithi186 ceph-mon[25720]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-22T05:18:46.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:46 smithi186 ceph-mon[25720]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-03-22T05:18:46.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:46 smithi186 ceph-mon[25720]: osdmap e45: 7 total, 6 up, 7 in 2024-03-22T05:18:46.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:46 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-22T05:18:46.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:46 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-22T05:18:46.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:46 smithi005 ceph-mon[24793]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-22T05:18:46.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:46 smithi005 ceph-mon[24793]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-03-22T05:18:46.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:46 smithi005 ceph-mon[24793]: osdmap e45: 7 total, 6 up, 7 in 2024-03-22T05:18:46.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:46 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-22T05:18:46.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:46 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-22T05:18:46.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:46 smithi005 ceph-mon[29523]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-22T05:18:46.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:46 smithi005 ceph-mon[29523]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-03-22T05:18:46.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:46 smithi005 ceph-mon[29523]: osdmap e45: 7 total, 6 up, 7 in 2024-03-22T05:18:46.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:46 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-22T05:18:46.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:46 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-22T05:18:47.679 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:47 smithi186 ceph-mon[25720]: purged_snaps scrub starts 2024-03-22T05:18:47.679 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:47 smithi186 ceph-mon[25720]: purged_snaps scrub ok 2024-03-22T05:18:47.679 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:47 smithi186 ceph-mon[25720]: osd.6 [v2:172.21.15.186:6816/2574435174,v1:172.21.15.186:6817/2574435174] boot 2024-03-22T05:18:47.680 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:47 smithi186 ceph-mon[25720]: osdmap e46: 7 total, 7 up, 7 in 2024-03-22T05:18:47.680 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:47 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-22T05:18:47.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:47 smithi005 ceph-mon[24793]: purged_snaps scrub starts 2024-03-22T05:18:47.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:47 smithi005 ceph-mon[24793]: purged_snaps scrub ok 2024-03-22T05:18:47.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:47 smithi005 ceph-mon[24793]: osd.6 [v2:172.21.15.186:6816/2574435174,v1:172.21.15.186:6817/2574435174] boot 2024-03-22T05:18:47.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:47 smithi005 ceph-mon[24793]: osdmap e46: 7 total, 7 up, 7 in 2024-03-22T05:18:47.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:47 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-22T05:18:47.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:47 smithi005 ceph-mon[29523]: purged_snaps scrub starts 2024-03-22T05:18:47.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:47 smithi005 ceph-mon[29523]: purged_snaps scrub ok 2024-03-22T05:18:47.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:47 smithi005 ceph-mon[29523]: osd.6 [v2:172.21.15.186:6816/2574435174,v1:172.21.15.186:6817/2574435174] boot 2024-03-22T05:18:47.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:47 smithi005 ceph-mon[29523]: osdmap e46: 7 total, 7 up, 7 in 2024-03-22T05:18:47.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:47 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-22T05:18:48.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:48 smithi005 ceph-mon[24793]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-22T05:18:48.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:48 smithi005 ceph-mon[24793]: osdmap e47: 7 total, 7 up, 7 in 2024-03-22T05:18:48.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:48 smithi005 ceph-mon[24793]: Detected new or changed devices on smithi186 2024-03-22T05:18:48.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:48 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:18:48.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:48 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:18:48.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:48 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:18:48.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:48 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:18:48.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:48 smithi005 ceph-mon[24793]: Adjusting osd_memory_target on smithi186 to 5696M 2024-03-22T05:18:48.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:48 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:18:48.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:48 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:18:48.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:48 smithi005 ceph-mon[29523]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-22T05:18:48.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:48 smithi005 ceph-mon[29523]: osdmap e47: 7 total, 7 up, 7 in 2024-03-22T05:18:48.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:48 smithi005 ceph-mon[29523]: Detected new or changed devices on smithi186 2024-03-22T05:18:48.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:48 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:18:48.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:48 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:18:48.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:48 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:18:48.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:48 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:18:48.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:48 smithi005 ceph-mon[29523]: Adjusting osd_memory_target on smithi186 to 5696M 2024-03-22T05:18:48.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:48 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:18:48.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:48 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:18:49.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:48 smithi186 ceph-mon[25720]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-22T05:18:49.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:48 smithi186 ceph-mon[25720]: osdmap e47: 7 total, 7 up, 7 in 2024-03-22T05:18:49.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:48 smithi186 ceph-mon[25720]: Detected new or changed devices on smithi186 2024-03-22T05:18:49.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:48 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:18:49.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:48 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:18:49.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:48 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:18:49.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:48 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:18:49.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:48 smithi186 ceph-mon[25720]: Adjusting osd_memory_target on smithi186 to 5696M 2024-03-22T05:18:49.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:48 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:18:49.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:48 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:18:49.050 INFO:teuthology.orchestra.run.smithi186.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-22T05:18:49.050 INFO:teuthology.orchestra.run.smithi186.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-22T05:18:49.050 INFO:teuthology.orchestra.run.smithi186.stderr: stderr: 10+0 records in 2024-03-22T05:18:49.050 INFO:teuthology.orchestra.run.smithi186.stderr:10+0 records out 2024-03-22T05:18:49.050 INFO:teuthology.orchestra.run.smithi186.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0124918 s, 839 MB/s 2024-03-22T05:18:49.051 INFO:teuthology.orchestra.run.smithi186.stderr:--> Zapping successful for: 2024-03-22T05:18:49.588 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:49 smithi186 ceph-mon[25720]: osdmap e48: 7 total, 7 up, 7 in 2024-03-22T05:18:49.613 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi186:vg_nvme/lv_1 2024-03-22T05:18:49.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:49 smithi005 ceph-mon[24793]: osdmap e48: 7 total, 7 up, 7 in 2024-03-22T05:18:49.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:49 smithi005 ceph-mon[29523]: osdmap e48: 7 total, 7 up, 7 in 2024-03-22T05:18:50.911 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:50 smithi186 ceph-mon[25720]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-03-22T05:18:50.911 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:50 smithi186 ceph-mon[25720]: osdmap e49: 7 total, 7 up, 7 in 2024-03-22T05:18:50.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:50 smithi005 ceph-mon[24793]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-03-22T05:18:50.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:50 smithi005 ceph-mon[24793]: osdmap e49: 7 total, 7 up, 7 in 2024-03-22T05:18:50.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:50 smithi005 ceph-mon[29523]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-03-22T05:18:50.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:50 smithi005 ceph-mon[29523]: osdmap e49: 7 total, 7 up, 7 in 2024-03-22T05:18:51.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:51 smithi005 ceph-mon[24793]: osdmap e50: 7 total, 7 up, 7 in 2024-03-22T05:18:51.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:51 smithi005 ceph-mon[29523]: osdmap e50: 7 total, 7 up, 7 in 2024-03-22T05:18:52.040 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:51 smithi186 ceph-mon[25720]: osdmap e50: 7 total, 7 up, 7 in 2024-03-22T05:18:52.695 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:52 smithi186 ceph-mon[25720]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-03-22T05:18:52.696 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:52 smithi186 ceph-mon[25720]: from='client.34283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:18:52.696 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:52 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T05:18:52.696 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:52 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T05:18:52.696 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:52 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:18:52.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:52 smithi005 ceph-mon[24793]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-03-22T05:18:52.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:52 smithi005 ceph-mon[24793]: from='client.34283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:18:52.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:52 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T05:18:52.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:52 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T05:18:52.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:52 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:18:52.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:52 smithi005 ceph-mon[29523]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-03-22T05:18:52.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:52 smithi005 ceph-mon[29523]: from='client.34283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:18:52.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:52 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-22T05:18:52.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:52 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-22T05:18:52.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:52 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:18:54.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:54 smithi005 ceph-mon[24793]: from='client.? 172.21.15.186:0/136673713' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0ae28e60-285b-423e-b686-f536cffbd595"}]: dispatch 2024-03-22T05:18:54.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:54 smithi005 ceph-mon[24793]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0ae28e60-285b-423e-b686-f536cffbd595"}]: dispatch 2024-03-22T05:18:54.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:54 smithi005 ceph-mon[24793]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0ae28e60-285b-423e-b686-f536cffbd595"}]': finished 2024-03-22T05:18:54.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:54 smithi005 ceph-mon[24793]: osdmap e51: 8 total, 7 up, 8 in 2024-03-22T05:18:54.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:54 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-22T05:18:54.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:54 smithi005 ceph-mon[29523]: from='client.? 172.21.15.186:0/136673713' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0ae28e60-285b-423e-b686-f536cffbd595"}]: dispatch 2024-03-22T05:18:54.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:54 smithi005 ceph-mon[29523]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0ae28e60-285b-423e-b686-f536cffbd595"}]: dispatch 2024-03-22T05:18:54.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:54 smithi005 ceph-mon[29523]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0ae28e60-285b-423e-b686-f536cffbd595"}]': finished 2024-03-22T05:18:54.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:54 smithi005 ceph-mon[29523]: osdmap e51: 8 total, 7 up, 8 in 2024-03-22T05:18:54.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:54 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-22T05:18:54.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:54 smithi186 ceph-mon[25720]: from='client.? 172.21.15.186:0/136673713' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0ae28e60-285b-423e-b686-f536cffbd595"}]: dispatch 2024-03-22T05:18:54.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:54 smithi186 ceph-mon[25720]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0ae28e60-285b-423e-b686-f536cffbd595"}]: dispatch 2024-03-22T05:18:54.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:54 smithi186 ceph-mon[25720]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0ae28e60-285b-423e-b686-f536cffbd595"}]': finished 2024-03-22T05:18:54.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:54 smithi186 ceph-mon[25720]: osdmap e51: 8 total, 7 up, 8 in 2024-03-22T05:18:54.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:54 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-22T05:18:55.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:55 smithi005 ceph-mon[24793]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-03-22T05:18:55.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:55 smithi005 ceph-mon[24793]: from='client.? 172.21.15.186:0/3378284531' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T05:18:55.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:55 smithi005 ceph-mon[29523]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-03-22T05:18:55.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:55 smithi005 ceph-mon[29523]: from='client.? 172.21.15.186:0/3378284531' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T05:18:55.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:55 smithi186 ceph-mon[25720]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-03-22T05:18:55.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:55 smithi186 ceph-mon[25720]: from='client.? 172.21.15.186:0/3378284531' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-22T05:18:57.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:57 smithi005 ceph-mon[24793]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-22T05:18:57.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:57 smithi005 ceph-mon[29523]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-22T05:18:57.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:57 smithi186 ceph-mon[25720]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-22T05:18:59.271 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:18:59 smithi186 ceph-mon[25720]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-03-22T05:18:59.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:18:59 smithi005 ceph-mon[24793]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-03-22T05:18:59.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:18:59 smithi005 ceph-mon[29523]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-03-22T05:19:01.224 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:01 smithi186 ceph-mon[25720]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-03-22T05:19:01.225 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:01 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-22T05:19:01.225 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:01 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:19:01.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:01 smithi005 ceph-mon[29523]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-03-22T05:19:01.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:01 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-22T05:19:01.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:01 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:19:01.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:01 smithi005 ceph-mon[24793]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-03-22T05:19:01.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:01 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-22T05:19:01.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:01 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:19:02.227 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:02 smithi186 ceph-mon[25720]: Deploying daemon osd.7 on smithi186 2024-03-22T05:19:02.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:02 smithi005 ceph-mon[24793]: Deploying daemon osd.7 on smithi186 2024-03-22T05:19:02.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:02 smithi005 ceph-mon[29523]: Deploying daemon osd.7 on smithi186 2024-03-22T05:19:03.248 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:03 smithi186 ceph-mon[25720]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-03-22T05:19:03.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:03 smithi005 ceph-mon[24793]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-03-22T05:19:03.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:03 smithi005 ceph-mon[29523]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-03-22T05:19:05.231 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:05 smithi186 ceph-mon[25720]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-22T05:19:05.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:05 smithi005 ceph-mon[24793]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-22T05:19:05.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:05 smithi005 ceph-mon[29523]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-22T05:19:07.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:06 smithi186 ceph-mon[25720]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-22T05:19:07.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:06 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:19:07.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:06 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:19:07.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:06 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:19:07.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:06 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:19:07.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:06 smithi005 ceph-mon[24793]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-22T05:19:07.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:06 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:19:07.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:06 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:19:07.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:06 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:19:07.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:06 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:19:07.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:06 smithi005 ceph-mon[29523]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-22T05:19:07.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:06 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:19:07.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:06 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:19:07.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:06 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:19:07.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:06 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:19:08.955 INFO:teuthology.orchestra.run.smithi186.stdout:Created osd(s) 7 on host 'smithi186' 2024-03-22T05:19:09.271 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:09 smithi186 ceph-mon[25720]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-22T05:19:09.272 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:09 smithi186 ceph-mon[25720]: from='osd.7 [v2:172.21.15.186:6824/3520054895,v1:172.21.15.186:6825/3520054895]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-22T05:19:09.272 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:09 smithi186 ceph-mon[25720]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-22T05:19:09.272 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:09 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:19:09.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:09 smithi005 ceph-mon[24793]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-22T05:19:09.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:09 smithi005 ceph-mon[24793]: from='osd.7 [v2:172.21.15.186:6824/3520054895,v1:172.21.15.186:6825/3520054895]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-22T05:19:09.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:09 smithi005 ceph-mon[24793]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-22T05:19:09.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:09 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:19:09.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:09 smithi005 ceph-mon[29523]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-22T05:19:09.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:09 smithi005 ceph-mon[29523]: from='osd.7 [v2:172.21.15.186:6824/3520054895,v1:172.21.15.186:6825/3520054895]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-22T05:19:09.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:09 smithi005 ceph-mon[29523]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-22T05:19:09.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:09 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:19:09.663 DEBUG:teuthology.orchestra.run.smithi186:osd.7> sudo journalctl -f -n 0 -u ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@osd.7.service 2024-03-22T05:19:09.666 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-03-22T05:19:09.666 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -- ceph osd stat -f json 2024-03-22T05:19:10.276 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:10 smithi005 ceph-mon[24793]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-22T05:19:10.276 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:10 smithi005 ceph-mon[24793]: osdmap e52: 8 total, 7 up, 8 in 2024-03-22T05:19:10.276 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:10 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-22T05:19:10.276 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:10 smithi005 ceph-mon[24793]: from='osd.7 [v2:172.21.15.186:6824/3520054895,v1:172.21.15.186:6825/3520054895]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-03-22T05:19:10.276 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:10 smithi005 ceph-mon[24793]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-03-22T05:19:10.276 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:10 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-22T05:19:10.276 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:10 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-22T05:19:10.276 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:10 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:19:10.276 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:10 smithi005 ceph-mon[29523]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-22T05:19:10.277 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:10 smithi005 ceph-mon[29523]: osdmap e52: 8 total, 7 up, 8 in 2024-03-22T05:19:10.277 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:10 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-22T05:19:10.277 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:10 smithi005 ceph-mon[29523]: from='osd.7 [v2:172.21.15.186:6824/3520054895,v1:172.21.15.186:6825/3520054895]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-03-22T05:19:10.277 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:10 smithi005 ceph-mon[29523]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-03-22T05:19:10.277 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:10 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-22T05:19:10.277 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:10 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-22T05:19:10.277 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:10 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:19:10.425 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:10 smithi186 ceph-mon[25720]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-22T05:19:10.426 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:10 smithi186 ceph-mon[25720]: osdmap e52: 8 total, 7 up, 8 in 2024-03-22T05:19:10.426 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:10 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-22T05:19:10.426 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:10 smithi186 ceph-mon[25720]: from='osd.7 [v2:172.21.15.186:6824/3520054895,v1:172.21.15.186:6825/3520054895]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-03-22T05:19:10.426 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:10 smithi186 ceph-mon[25720]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-03-22T05:19:10.426 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:10 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-22T05:19:10.426 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:10 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-22T05:19:10.426 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:10 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:19:10.426 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:19:10 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[37870]: 2024-03-22T05:19:10.128+0000 7f39615f9700 -1 osd.7 0 waiting for initial osdmap 2024-03-22T05:19:10.426 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:19:10 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[37870]: 2024-03-22T05:19:10.134+0000 7f3959f8c700 -1 osd.7 53 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-22T05:19:11.286 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:11 smithi005 ceph-mon[29523]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-22T05:19:11.286 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:11 smithi005 ceph-mon[29523]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-03-22T05:19:11.286 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:11 smithi005 ceph-mon[29523]: osdmap e53: 8 total, 7 up, 8 in 2024-03-22T05:19:11.286 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:11 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-22T05:19:11.286 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:11 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-22T05:19:11.287 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:11 smithi005 ceph-mon[24793]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-22T05:19:11.287 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:11 smithi005 ceph-mon[24793]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-03-22T05:19:11.287 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:11 smithi005 ceph-mon[24793]: osdmap e53: 8 total, 7 up, 8 in 2024-03-22T05:19:11.287 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:11 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-22T05:19:11.287 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:11 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-22T05:19:11.375 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:11 smithi186 ceph-mon[25720]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-22T05:19:11.375 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:11 smithi186 ceph-mon[25720]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-03-22T05:19:11.375 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:11 smithi186 ceph-mon[25720]: osdmap e53: 8 total, 7 up, 8 in 2024-03-22T05:19:11.375 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:11 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-22T05:19:11.375 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:11 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-22T05:19:11.707 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-22T05:19:12.386 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:12 smithi005 ceph-mon[29523]: purged_snaps scrub starts 2024-03-22T05:19:12.386 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:12 smithi005 ceph-mon[29523]: purged_snaps scrub ok 2024-03-22T05:19:12.386 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:12 smithi005 ceph-mon[29523]: osd.7 [v2:172.21.15.186:6824/3520054895,v1:172.21.15.186:6825/3520054895] boot 2024-03-22T05:19:12.386 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:12 smithi005 ceph-mon[29523]: osdmap e54: 8 total, 8 up, 8 in 2024-03-22T05:19:12.386 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:12 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-22T05:19:12.386 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:12 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/1378635199' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-22T05:19:12.386 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:12 smithi005 ceph-mon[29523]: osdmap e55: 8 total, 8 up, 8 in 2024-03-22T05:19:12.387 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:12 smithi005 ceph-mon[24793]: purged_snaps scrub starts 2024-03-22T05:19:12.387 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:12 smithi005 ceph-mon[24793]: purged_snaps scrub ok 2024-03-22T05:19:12.387 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:12 smithi005 ceph-mon[24793]: osd.7 [v2:172.21.15.186:6824/3520054895,v1:172.21.15.186:6825/3520054895] boot 2024-03-22T05:19:12.387 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:12 smithi005 ceph-mon[24793]: osdmap e54: 8 total, 8 up, 8 in 2024-03-22T05:19:12.387 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:12 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-22T05:19:12.387 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:12 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/1378635199' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-22T05:19:12.387 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:12 smithi005 ceph-mon[24793]: osdmap e55: 8 total, 8 up, 8 in 2024-03-22T05:19:12.411 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":54,"num_osds":8,"num_up_osds":8,"osd_up_since":1711084751,"num_in_osds":8,"osd_in_since":1711084733,"num_remapped_pgs":1} 2024-03-22T05:19:12.412 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-22T05:19:12.468 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:12 smithi186 ceph-mon[25720]: purged_snaps scrub starts 2024-03-22T05:19:12.468 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:12 smithi186 ceph-mon[25720]: purged_snaps scrub ok 2024-03-22T05:19:12.468 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:12 smithi186 ceph-mon[25720]: osd.7 [v2:172.21.15.186:6824/3520054895,v1:172.21.15.186:6825/3520054895] boot 2024-03-22T05:19:12.468 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:12 smithi186 ceph-mon[25720]: osdmap e54: 8 total, 8 up, 8 in 2024-03-22T05:19:12.468 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:12 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-22T05:19:12.469 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:12 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/1378635199' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-22T05:19:12.469 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:12 smithi186 ceph-mon[25720]: osdmap e55: 8 total, 8 up, 8 in 2024-03-22T05:19:12.643 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T05:19:13.401 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:13 smithi005 ceph-mon[24793]: pgmap v165: 1 pgs: 1 peering; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-22T05:19:13.401 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:13 smithi005 ceph-mon[24793]: Detected new or changed devices on smithi186 2024-03-22T05:19:13.401 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:13 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:19:13.401 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:13 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:19:13.401 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:13 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:19:13.401 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:13 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:19:13.401 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:13 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:19:13.401 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:13 smithi005 ceph-mon[24793]: Adjusting osd_memory_target on smithi186 to 4272M 2024-03-22T05:19:13.401 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:13 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:19:13.401 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:13 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:19:13.401 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:13 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:19:13.401 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:13 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:19:13.402 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:13 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:19:13.402 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:13 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:19:13.402 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:13 smithi005 ceph-mon[24793]: osdmap e56: 8 total, 8 up, 8 in 2024-03-22T05:19:13.402 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:13 smithi005 ceph-mon[29523]: pgmap v165: 1 pgs: 1 peering; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-22T05:19:13.402 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:13 smithi005 ceph-mon[29523]: Detected new or changed devices on smithi186 2024-03-22T05:19:13.402 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:13 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:19:13.402 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:13 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:19:13.402 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:13 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:19:13.402 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:13 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:19:13.402 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:13 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:19:13.402 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:13 smithi005 ceph-mon[29523]: Adjusting osd_memory_target on smithi186 to 4272M 2024-03-22T05:19:13.402 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:13 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:19:13.403 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:13 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:19:13.403 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:13 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:19:13.403 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:13 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:19:13.403 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:13 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:19:13.403 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:13 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:19:13.403 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:13 smithi005 ceph-mon[29523]: osdmap e56: 8 total, 8 up, 8 in 2024-03-22T05:19:13.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:13 smithi186 ceph-mon[25720]: pgmap v165: 1 pgs: 1 peering; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-22T05:19:13.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:13 smithi186 ceph-mon[25720]: Detected new or changed devices on smithi186 2024-03-22T05:19:13.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:13 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:19:13.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:13 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:19:13.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:13 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:19:13.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:13 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:19:13.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:13 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:19:13.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:13 smithi186 ceph-mon[25720]: Adjusting osd_memory_target on smithi186 to 4272M 2024-03-22T05:19:13.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:13 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:19:13.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:13 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:19:13.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:13 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:19:13.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:13 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:19:13.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:13 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:19:13.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:13 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:19:13.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:13 smithi186 ceph-mon[25720]: osdmap e56: 8 total, 8 up, 8 in 2024-03-22T05:19:14.296 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:14 smithi005 ceph-mon[24793]: pgmap v168: 1 pgs: 1 peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:19:14.296 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:14 smithi005 ceph-mon[29523]: pgmap v168: 1 pgs: 1 peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:19:14.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:14 smithi186 ceph-mon[25720]: pgmap v168: 1 pgs: 1 peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:19:14.584 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-22T05:19:14.584 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":57,"fsid":"f59f36e2-e80a-11ee-95cd-87774f69a715","created":"2024-03-22T05:14:18.934387+0000","modified":"2024-03-22T05:19:14.143018+0000","last_up_change":"2024-03-22T05:19:11.125488+0000","last_in_change":"2024-03-22T05:18:53.961201+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-22T05:17:17.465016+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"a0063741-df11-4674-9d0e-0ac5161bd1b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":2350146241},{"type":"v1","addr":"172.21.15.5:6803","nonce":2350146241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":2350146241},{"type":"v1","addr":"172.21.15.5:6805","nonce":2350146241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":2350146241},{"type":"v1","addr":"172.21.15.5:6809","nonce":2350146241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":2350146241},{"type":"v1","addr":"172.21.15.5:6807","nonce":2350146241}]},"public_addr":"172.21.15.5:6803/2350146241","cluster_addr":"172.21.15.5:6805/2350146241","heartbeat_back_addr":"172.21.15.5:6809/2350146241","heartbeat_front_addr":"172.21.15.5:6807/2350146241","state":["exists","up"]},{"osd":1,"uuid":"ef4e89fc-61a9-46c7-8357-2378e58055b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6810","nonce":790214800},{"type":"v1","addr":"172.21.15.5:6811","nonce":790214800}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":790214800},{"type":"v1","addr":"172.21.15.5:6813","nonce":790214800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6816","nonce":790214800},{"type":"v1","addr":"172.21.15.5:6817","nonce":790214800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":790214800},{"type":"v1","addr":"172.21.15.5:6815","nonce":790214800}]},"public_addr":"172.21.15.5:6811/790214800","cluster_addr":"172.21.15.5:6813/790214800","heartbeat_back_addr":"172.21.15.5:6817/790214800","heartbeat_front_addr":"172.21.15.5:6815/790214800","state":["exists","up"]},{"osd":2,"uuid":"c1365858-72bc-4a3a-84cb-ac8f1beaa725","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":3354758722},{"type":"v1","addr":"172.21.15.5:6819","nonce":3354758722}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":3354758722},{"type":"v1","addr":"172.21.15.5:6821","nonce":3354758722}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":3354758722},{"type":"v1","addr":"172.21.15.5:6825","nonce":3354758722}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6822","nonce":3354758722},{"type":"v1","addr":"172.21.15.5:6823","nonce":3354758722}]},"public_addr":"172.21.15.5:6819/3354758722","cluster_addr":"172.21.15.5:6821/3354758722","heartbeat_back_addr":"172.21.15.5:6825/3354758722","heartbeat_front_addr":"172.21.15.5:6823/3354758722","state":["exists","up"]},{"osd":3,"uuid":"ae0fa9b1-b54d-4c9a-b5d0-33ae91dcf98d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":3829723611},{"type":"v1","addr":"172.21.15.5:6827","nonce":3829723611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6828","nonce":3829723611},{"type":"v1","addr":"172.21.15.5:6829","nonce":3829723611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":3829723611},{"type":"v1","addr":"172.21.15.5:6833","nonce":3829723611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":3829723611},{"type":"v1","addr":"172.21.15.5:6831","nonce":3829723611}]},"public_addr":"172.21.15.5:6827/3829723611","cluster_addr":"172.21.15.5:6829/3829723611","heartbeat_back_addr":"172.21.15.5:6833/3829723611","heartbeat_front_addr":"172.21.15.5:6831/3829723611","state":["exists","up"]},{"osd":4,"uuid":"7313be61-8567-42c4-93a0-8ff3f2e30da3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":2861669931},{"type":"v1","addr":"172.21.15.186:6801","nonce":2861669931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":2861669931},{"type":"v1","addr":"172.21.15.186:6803","nonce":2861669931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":2861669931},{"type":"v1","addr":"172.21.15.186:6807","nonce":2861669931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":2861669931},{"type":"v1","addr":"172.21.15.186:6805","nonce":2861669931}]},"public_addr":"172.21.15.186:6801/2861669931","cluster_addr":"172.21.15.186:6803/2861669931","heartbeat_back_addr":"172.21.15.186:6807/2861669931","heartbeat_front_addr":"172.21.15.186:6805/2861669931","state":["exists","up"]},{"osd":5,"uuid":"edeed5ee-7262-4ce5-a32f-cef71b2d6acf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":554926692},{"type":"v1","addr":"172.21.15.186:6809","nonce":554926692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":554926692},{"type":"v1","addr":"172.21.15.186:6811","nonce":554926692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":554926692},{"type":"v1","addr":"172.21.15.186:6815","nonce":554926692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":554926692},{"type":"v1","addr":"172.21.15.186:6813","nonce":554926692}]},"public_addr":"172.21.15.186:6809/554926692","cluster_addr":"172.21.15.186:6811/554926692","heartbeat_back_addr":"172.21.15.186:6815/554926692","heartbeat_front_addr":"172.21.15.186:6813/554926692","state":["exists","up"]},{"osd":6,"uuid":"b979e26c-6b94-434f-8c95-9b9a8713eb78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":2574435174},{"type":"v1","addr":"172.21.15.186:6817","nonce":2574435174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":2574435174},{"type":"v1","addr":"172.21.15.186:6819","nonce":2574435174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":2574435174},{"type":"v1","addr":"172.21.15.186:6823","nonce":2574435174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":2574435174},{"type":"v1","addr":"172.21.15.186:6821","nonce":2574435174}]},"public_addr":"172.21.15.186:6817/2574435174","cluster_addr":"172.21.15.186:6819/2574435174","heartbeat_back_addr":"172.21.15.186:6823/2574435174","heartbeat_front_addr":"172.21.15.186:6821/2574435174","state":["exists","up"]},{"osd":7,"uuid":"0ae28e60-285b-423e-b686-f536cffbd595","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3520054895},{"type":"v1","addr":"172.21.15.186:6825","nonce":3520054895}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3520054895},{"type":"v1","addr":"172.21.15.186:6827","nonce":3520054895}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3520054895},{"type":"v1","addr":"172.21.15.186:6831","nonce":3520054895}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3520054895},{"type":"v1","addr":"172.21.15.186:6829","nonce":3520054895}]},"public_addr":"172.21.15.186:6825/3520054895","cluster_addr":"172.21.15.186:6827/3520054895","heartbeat_back_addr":"172.21.15.186:6831/3520054895","heartbeat_front_addr":"172.21.15.186:6829/3520054895","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T05:16:28.343379+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T05:16:51.136049+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T05:17:15.010413+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T05:17:38.810252+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T05:17:59.769820+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T05:18:21.904758+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T05:18:45.000458+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T05:19:09.910476+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.5:0/2650789828":"2024-03-23T05:15:09.085731+0000","172.21.15.5:6801/2160302520":"2024-03-23T05:15:09.085731+0000","172.21.15.5:6800/2160302520":"2024-03-23T05:15:09.085731+0000","172.21.15.5:0/2169353506":"2024-03-23T05:15:09.085731+0000","172.21.15.5:0/3301532190":"2024-03-23T05:14:47.571432+0000","172.21.15.5:0/2858060116":"2024-03-23T05:14:47.571432+0000","172.21.15.5:6800/392664475":"2024-03-23T05:14:47.571432+0000","172.21.15.5:6801/392664475":"2024-03-23T05:14:47.571432+0000","172.21.15.5:0/2331329527":"2024-03-23T05:15:09.085731+0000","172.21.15.5:0/789815545":"2024-03-23T05:14:47.571432+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-22T05:19:15.258 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:15 smithi005 ceph-mon[24793]: osdmap e57: 8 total, 8 up, 8 in 2024-03-22T05:19:15.259 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:15 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/1304830369' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T05:19:15.259 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:15 smithi005 ceph-mon[29523]: osdmap e57: 8 total, 8 up, 8 in 2024-03-22T05:19:15.259 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:15 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/1304830369' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T05:19:15.285 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-22T05:17:17.465016+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-03-22T05:19:15.286 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-22T05:19:15.508 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T05:19:15.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:15 smithi186 ceph-mon[25720]: osdmap e57: 8 total, 8 up, 8 in 2024-03-22T05:19:15.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:15 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/1304830369' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T05:19:16.306 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:16 smithi005 ceph-mon[24793]: osdmap e58: 8 total, 8 up, 8 in 2024-03-22T05:19:16.307 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:16 smithi005 ceph-mon[24793]: pgmap v171: 1 pgs: 1 peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:19:16.307 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:16 smithi005 ceph-mon[29523]: osdmap e58: 8 total, 8 up, 8 in 2024-03-22T05:19:16.307 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:16 smithi005 ceph-mon[29523]: pgmap v171: 1 pgs: 1 peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:19:16.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:16 smithi186 ceph-mon[25720]: osdmap e58: 8 total, 8 up, 8 in 2024-03-22T05:19:16.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:16 smithi186 ceph-mon[25720]: pgmap v171: 1 pgs: 1 peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:19:17.280 INFO:teuthology.orchestra.run.smithi005.stdout:pg_num: 1 2024-03-22T05:19:18.032 INFO:tasks.cephadm:Adding prometheus.a on smithi186 2024-03-22T05:19:18.032 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -- ceph orch apply prometheus '1;smithi186=a' 2024-03-22T05:19:18.352 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:18 smithi186 ceph-mon[25720]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-03-22T05:19:18.352 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:18 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/3935275926' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T05:19:18.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:18 smithi005 ceph-mon[24793]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-03-22T05:19:18.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:18 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/3935275926' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T05:19:18.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:18 smithi005 ceph-mon[29523]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-03-22T05:19:18.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:18 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/3935275926' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-22T05:19:19.976 INFO:teuthology.orchestra.run.smithi186.stdout:Scheduled prometheus update... 2024-03-22T05:19:20.529 DEBUG:teuthology.orchestra.run.smithi186:prometheus.a> sudo journalctl -f -n 0 -u ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@prometheus.a.service 2024-03-22T05:19:20.531 INFO:tasks.cephadm:Adding node-exporter.a on smithi005 2024-03-22T05:19:20.531 INFO:tasks.cephadm:Adding node-exporter.b on smithi186 2024-03-22T05:19:20.531 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -- ceph orch apply node-exporter '2;smithi005=a;smithi186=b' 2024-03-22T05:19:21.028 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:20 smithi186 ceph-mon[25720]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-03-22T05:19:21.028 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:20 smithi186 ceph-mon[25720]: from='client.34325 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi186=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:19:21.028 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:20 smithi186 ceph-mon[25720]: Saving service prometheus spec with placement smithi186=a;count:1 2024-03-22T05:19:21.028 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:20 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:19:21.028 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:20 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:19:21.028 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:20 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:19:21.028 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:20 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:19:21.028 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:20 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:19:21.028 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:20 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-22T05:19:21.294 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:19:21 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: ignoring --setuser ceph since I am not root 2024-03-22T05:19:21.294 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:19:21 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: ignoring --setgroup ceph since I am not root 2024-03-22T05:19:21.294 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:19:21 smithi186 ceph-mgr[26894]: -- 172.21.15.186:0/3603363589 <== mon.2 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x562e1fcf6340 con 0x562e1fccf000 2024-03-22T05:19:21.452 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:19:21 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: ignoring --setuser ceph since I am not root 2024-03-22T05:19:21.452 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:19:21 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: ignoring --setgroup ceph since I am not root 2024-03-22T05:19:21.452 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:19:21 smithi005 ceph-mgr[25025]: -- 172.21.15.5:0/2277158407 <== mon.1 v2:172.21.15.5:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55695d40e340 con 0x55695d3e6c00 2024-03-22T05:19:21.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:20 smithi005 ceph-mon[29523]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-03-22T05:19:21.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:20 smithi005 ceph-mon[29523]: from='client.34325 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi186=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:19:21.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:20 smithi005 ceph-mon[29523]: Saving service prometheus spec with placement smithi186=a;count:1 2024-03-22T05:19:21.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:20 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:19:21.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:20 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:19:21.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:20 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:19:21.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:20 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:19:21.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:20 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:19:21.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:20 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-22T05:19:21.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:20 smithi005 ceph-mon[24793]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-03-22T05:19:21.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:20 smithi005 ceph-mon[24793]: from='client.34325 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi186=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:19:21.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:20 smithi005 ceph-mon[24793]: Saving service prometheus spec with placement smithi186=a;count:1 2024-03-22T05:19:21.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:20 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:19:21.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:20 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:19:21.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:20 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:19:21.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:20 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:19:21.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:20 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' 2024-03-22T05:19:21.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:20 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-22T05:19:21.793 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:19:21 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: 2024-03-22T05:19:21.522+0000 7f3562524000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-22T05:19:21.794 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:19:21 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: 2024-03-22T05:19:21.613+0000 7f3562524000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-22T05:19:21.954 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:19:21 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:19:21.514+0000 7f1bfbe3b000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-22T05:19:21.954 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:19:21 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:19:21.605+0000 7f1bfbe3b000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-22T05:19:22.223 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:21 smithi186 ceph-mon[25720]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-22T05:19:22.223 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:21 smithi186 ceph-mon[25720]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-22T05:19:22.300 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:19:22 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:19:22.209+0000 7f1bfbe3b000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-22T05:19:22.301 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:21 smithi005 ceph-mon[24793]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-22T05:19:22.301 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:21 smithi005 ceph-mon[24793]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-22T05:19:22.301 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:21 smithi005 ceph-mon[29523]: from='mgr.14150 172.21.15.5:0/932697351' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-22T05:19:22.301 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:21 smithi005 ceph-mon[29523]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-22T05:19:22.543 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:19:22 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: 2024-03-22T05:19:22.222+0000 7f3562524000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-22T05:19:22.544 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:19:22 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: 2024-03-22T05:19:22.314+0000 7f3562524000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-22T05:19:22.544 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:19:22 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: 2024-03-22T05:19:22.400+0000 7f3562524000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-22T05:19:22.619 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:19:22 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:19:22.300+0000 7f1bfbe3b000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-22T05:19:22.619 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:19:22 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:19:22.386+0000 7f1bfbe3b000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-22T05:19:22.620 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:19:22 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:19:22.537+0000 7f1bfbe3b000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-22T05:19:22.620 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:19:22 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:19:22.619+0000 7f1bfbe3b000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-22T05:19:22.829 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:19:22 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: 2024-03-22T05:19:22.556+0000 7f3562524000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-22T05:19:22.829 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:19:22 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: 2024-03-22T05:19:22.636+0000 7f3562524000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-22T05:19:22.830 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:19:22 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: 2024-03-22T05:19:22.746+0000 7f3562524000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-22T05:19:22.830 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:19:22 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: 2024-03-22T05:19:22.829+0000 7f3562524000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-22T05:19:22.952 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:19:22 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:19:22.731+0000 7f1bfbe3b000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-22T05:19:22.952 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:19:22 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:19:22.816+0000 7f1bfbe3b000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-22T05:19:23.293 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:19:22 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: 2024-03-22T05:19:22.991+0000 7f3562524000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-22T05:19:23.294 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:19:23 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: 2024-03-22T05:19:23.075+0000 7f3562524000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-22T05:19:23.452 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:19:22 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:19:22.981+0000 7f1bfbe3b000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-22T05:19:23.452 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:19:23 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:19:23.073+0000 7f1bfbe3b000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-22T05:19:24.414 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:19:24 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: 2024-03-22T05:19:24.098+0000 7f3562524000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-22T05:19:24.414 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:19:24 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: 2024-03-22T05:19:24.226+0000 7f3562524000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-22T05:19:24.415 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:19:24 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: 2024-03-22T05:19:24.414+0000 7f3562524000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-22T05:19:24.452 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:19:24 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:19:24.136+0000 7f1bfbe3b000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-22T05:19:24.452 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:19:24 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:19:24.269+0000 7f1bfbe3b000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-22T05:19:24.793 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:19:24 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: 2024-03-22T05:19:24.501+0000 7f3562524000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-22T05:19:24.794 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:19:24 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: 2024-03-22T05:19:24.636+0000 7f3562524000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-22T05:19:24.952 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:19:24 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:19:24.465+0000 7f1bfbe3b000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-22T05:19:24.952 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:19:24 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:19:24.554+0000 7f1bfbe3b000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-22T05:19:24.952 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:19:24 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:19:24.694+0000 7f1bfbe3b000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-22T05:19:25.258 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:19:24 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: 2024-03-22T05:19:24.955+0000 7f3562524000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-22T05:19:25.258 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:19:25 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: 2024-03-22T05:19:25.174+0000 7f3562524000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-22T05:19:25.259 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:19:25 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: 2024-03-22T05:19:25.257+0000 7f3562524000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-22T05:19:25.333 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:19:25 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:19:25.018+0000 7f1bfbe3b000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-22T05:19:25.333 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:19:25 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:19:25.246+0000 7f1bfbe3b000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-22T05:19:25.543 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:19:25 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: 2024-03-22T05:19:25.426+0000 7f3562524000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-22T05:19:25.702 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:19:25 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:19:25.333+0000 7f1bfbe3b000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-22T05:19:25.702 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:19:25 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:19:25.508+0000 7f1bfbe3b000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-22T05:19:26.630 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:19:26 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: 2024-03-22T05:19:26.337+0000 7f3562524000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-22T05:19:26.702 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:19:26 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:19:26.452+0000 7f1bfbe3b000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-22T05:19:26.889 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:19:26 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: 2024-03-22T05:19:26.630+0000 7f3562524000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-22T05:19:26.889 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:19:26 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: 2024-03-22T05:19:26.724+0000 7f3562524000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-22T05:19:26.889 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:19:26 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: 2024-03-22T05:19:26.873+0000 7f3562524000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-22T05:19:26.889 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:19:26 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: [22/Mar/2024:05:19:26] ENGINE Bus STARTING 2024-03-22T05:19:27.015 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:19:26 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:19:26.757+0000 7f1bfbe3b000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-22T05:19:27.015 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:19:26 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:19:26.856+0000 7f1bfbe3b000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-22T05:19:27.015 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:26 smithi005 ceph-mon[24793]: Standby manager daemon x restarted 2024-03-22T05:19:27.015 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:26 smithi005 ceph-mon[24793]: Standby manager daemon x started 2024-03-22T05:19:27.015 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:26 smithi005 ceph-mon[24793]: from='mgr.? 172.21.15.186:0/683149886' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-22T05:19:27.015 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:26 smithi005 ceph-mon[24793]: from='mgr.? 172.21.15.186:0/683149886' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-22T05:19:27.015 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:26 smithi005 ceph-mon[24793]: from='mgr.? 172.21.15.186:0/683149886' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-22T05:19:27.015 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:26 smithi005 ceph-mon[24793]: from='mgr.? 172.21.15.186:0/683149886' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-22T05:19:27.016 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:26 smithi005 ceph-mon[29523]: Standby manager daemon x restarted 2024-03-22T05:19:27.016 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:26 smithi005 ceph-mon[29523]: Standby manager daemon x started 2024-03-22T05:19:27.016 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:26 smithi005 ceph-mon[29523]: from='mgr.? 172.21.15.186:0/683149886' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-22T05:19:27.016 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:26 smithi005 ceph-mon[29523]: from='mgr.? 172.21.15.186:0/683149886' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-22T05:19:27.016 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:26 smithi005 ceph-mon[29523]: from='mgr.? 172.21.15.186:0/683149886' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-22T05:19:27.016 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:26 smithi005 ceph-mon[29523]: from='mgr.? 172.21.15.186:0/683149886' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-22T05:19:27.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:26 smithi186 ceph-mon[25720]: Standby manager daemon x restarted 2024-03-22T05:19:27.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:26 smithi186 ceph-mon[25720]: Standby manager daemon x started 2024-03-22T05:19:27.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:26 smithi186 ceph-mon[25720]: from='mgr.? 172.21.15.186:0/683149886' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-22T05:19:27.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:26 smithi186 ceph-mon[25720]: from='mgr.? 172.21.15.186:0/683149886' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-22T05:19:27.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:26 smithi186 ceph-mon[25720]: from='mgr.? 172.21.15.186:0/683149886' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-22T05:19:27.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:26 smithi186 ceph-mon[25720]: from='mgr.? 172.21.15.186:0/683149886' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-22T05:19:27.294 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:19:26 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: CherryPy Checker: 2024-03-22T05:19:27.294 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:19:26 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: The Application mounted at '' has an empty config. 2024-03-22T05:19:27.295 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:19:26 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: 2024-03-22T05:19:27.295 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:19:26 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: [22/Mar/2024:05:19:26] ENGINE Serving on http://:::9283 2024-03-22T05:19:27.295 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:19:26 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: [22/Mar/2024:05:19:26] ENGINE Bus STARTED 2024-03-22T05:19:27.452 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:19:27 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:19:27.015+0000 7f1bfbe3b000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-22T05:19:28.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:27 smithi005 ceph-mon[24793]: mgrmap e17: y(active, since 4m), standbys: x 2024-03-22T05:19:28.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:27 smithi005 ceph-mon[24793]: Active manager daemon y restarted 2024-03-22T05:19:28.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:27 smithi005 ceph-mon[24793]: Activating manager daemon y 2024-03-22T05:19:28.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:27 smithi005 ceph-mon[24793]: osdmap e59: 8 total, 8 up, 8 in 2024-03-22T05:19:28.203 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:19:28 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: [22/Mar/2024:05:19:28] ENGINE Bus STARTING 2024-03-22T05:19:28.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:27 smithi005 ceph-mon[29523]: mgrmap e17: y(active, since 4m), standbys: x 2024-03-22T05:19:28.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:27 smithi005 ceph-mon[29523]: Active manager daemon y restarted 2024-03-22T05:19:28.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:27 smithi005 ceph-mon[29523]: Activating manager daemon y 2024-03-22T05:19:28.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:27 smithi005 ceph-mon[29523]: osdmap e59: 8 total, 8 up, 8 in 2024-03-22T05:19:28.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:27 smithi186 ceph-mon[25720]: mgrmap e17: y(active, since 4m), standbys: x 2024-03-22T05:19:28.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:27 smithi186 ceph-mon[25720]: Active manager daemon y restarted 2024-03-22T05:19:28.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:27 smithi186 ceph-mon[25720]: Activating manager daemon y 2024-03-22T05:19:28.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:27 smithi186 ceph-mon[25720]: osdmap e59: 8 total, 8 up, 8 in 2024-03-22T05:19:28.580 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:19:28 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: CherryPy Checker: 2024-03-22T05:19:28.580 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:19:28 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: The Application mounted at '' has an empty config. 2024-03-22T05:19:28.580 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:19:28 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:19:28.580 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:19:28 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: [22/Mar/2024:05:19:28] ENGINE Serving on http://:::9283 2024-03-22T05:19:28.580 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:19:28 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: [22/Mar/2024:05:19:28] ENGINE Bus STARTED 2024-03-22T05:19:28.580 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:19:28 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: [22/Mar/2024:05:19:28] ENGINE Bus STARTING 2024-03-22T05:19:28.580 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:19:28 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: [22/Mar/2024:05:19:28] ENGINE Serving on https://172.21.15.5:7150 2024-03-22T05:19:28.580 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:19:28 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: [22/Mar/2024:05:19:28] ENGINE Bus STARTED 2024-03-22T05:19:28.971 INFO:teuthology.orchestra.run.smithi186.stdout:Scheduled node-exporter update... 2024-03-22T05:19:29.266 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:28 smithi005 ceph-mon[24793]: mgrmap e18: y(active, starting, since 0.926698s), standbys: x 2024-03-22T05:19:29.266 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:28 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-22T05:19:29.266 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:28 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T05:19:29.266 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:28 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T05:19:29.266 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:28 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-22T05:19:29.266 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:28 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-22T05:19:29.266 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:28 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T05:19:29.266 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:28 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T05:19:29.267 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:28 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T05:19:29.267 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:28 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T05:19:29.267 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:28 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T05:19:29.267 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:28 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T05:19:29.267 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:28 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-22T05:19:29.267 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:28 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-22T05:19:29.267 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:28 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-22T05:19:29.267 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:28 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-22T05:19:29.267 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:28 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-22T05:19:29.267 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:28 smithi005 ceph-mon[24793]: Manager daemon y is now available 2024-03-22T05:19:29.267 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:28 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:19:29.267 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:28 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:19:29.267 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:28 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:19:29.267 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:28 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:19:29.268 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:28 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:19:29.268 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:28 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-22T05:19:29.268 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:28 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-22T05:19:29.268 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:28 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:19:29.268 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:28 smithi005 ceph-mon[29523]: mgrmap e18: y(active, starting, since 0.926698s), standbys: x 2024-03-22T05:19:29.268 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:28 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-22T05:19:29.268 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:28 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T05:19:29.268 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:28 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T05:19:29.268 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:28 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-22T05:19:29.268 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:28 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-22T05:19:29.268 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:28 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T05:19:29.269 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:28 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T05:19:29.269 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:28 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T05:19:29.269 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:28 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T05:19:29.269 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:28 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T05:19:29.269 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:28 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T05:19:29.269 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:28 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-22T05:19:29.269 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:28 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-22T05:19:29.269 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:28 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-22T05:19:29.269 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:28 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-22T05:19:29.269 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:28 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-22T05:19:29.269 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:28 smithi005 ceph-mon[29523]: Manager daemon y is now available 2024-03-22T05:19:29.269 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:28 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:19:29.269 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:28 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:19:29.269 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:28 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:19:29.270 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:28 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:19:29.270 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:28 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:19:29.270 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:28 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-22T05:19:29.270 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:28 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-22T05:19:29.270 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:28 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:19:29.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:28 smithi186 ceph-mon[25720]: mgrmap e18: y(active, starting, since 0.926698s), standbys: x 2024-03-22T05:19:29.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:28 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-22T05:19:29.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:28 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T05:19:29.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:28 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T05:19:29.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:28 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-22T05:19:29.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:28 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-22T05:19:29.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:28 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T05:19:29.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:28 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T05:19:29.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:28 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T05:19:29.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:28 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T05:19:29.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:28 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T05:19:29.295 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:28 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T05:19:29.295 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:28 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-22T05:19:29.295 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:28 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-22T05:19:29.295 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:28 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-22T05:19:29.295 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:28 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-22T05:19:29.295 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:28 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-22T05:19:29.295 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:28 smithi186 ceph-mon[25720]: Manager daemon y is now available 2024-03-22T05:19:29.295 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:28 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:19:29.295 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:28 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:19:29.295 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:28 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:19:29.295 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:28 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:19:29.295 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:28 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:19:29.295 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:28 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-22T05:19:29.295 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:28 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-22T05:19:29.295 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:28 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:19:29.853 DEBUG:teuthology.orchestra.run.smithi005:node-exporter.a> sudo journalctl -f -n 0 -u ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@node-exporter.a.service 2024-03-22T05:19:29.856 DEBUG:teuthology.orchestra.run.smithi186:node-exporter.b> sudo journalctl -f -n 0 -u ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@node-exporter.b.service 2024-03-22T05:19:29.858 INFO:tasks.cephadm:Adding alertmanager.a on smithi005 2024-03-22T05:19:29.859 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -- ceph orch apply alertmanager '1;smithi005=a' 2024-03-22T05:19:30.121 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:29 smithi186 ceph-mon[25720]: [22/Mar/2024:05:19:28] ENGINE Bus STARTING 2024-03-22T05:19:30.121 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:29 smithi186 ceph-mon[25720]: [22/Mar/2024:05:19:28] ENGINE Serving on https://172.21.15.5:7150 2024-03-22T05:19:30.121 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:29 smithi186 ceph-mon[25720]: [22/Mar/2024:05:19:28] ENGINE Bus STARTED 2024-03-22T05:19:30.121 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:29 smithi186 ceph-mon[25720]: mgrmap e19: y(active, since 1.9302s), standbys: x 2024-03-22T05:19:30.121 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:29 smithi186 ceph-mon[25720]: from='client.34343 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi005=a;smithi186=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:19:30.121 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:29 smithi186 ceph-mon[25720]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:19:30.121 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:29 smithi186 ceph-mon[25720]: Saving service node-exporter spec with placement smithi005=a;smithi186=b;count:2 2024-03-22T05:19:30.121 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:29 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:19:30.121 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:29 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:19:30.285 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:29 smithi005 ceph-mon[29523]: [22/Mar/2024:05:19:28] ENGINE Bus STARTING 2024-03-22T05:19:30.285 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:29 smithi005 ceph-mon[29523]: [22/Mar/2024:05:19:28] ENGINE Serving on https://172.21.15.5:7150 2024-03-22T05:19:30.285 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:29 smithi005 ceph-mon[29523]: [22/Mar/2024:05:19:28] ENGINE Bus STARTED 2024-03-22T05:19:30.285 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:29 smithi005 ceph-mon[29523]: mgrmap e19: y(active, since 1.9302s), standbys: x 2024-03-22T05:19:30.285 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:29 smithi005 ceph-mon[29523]: from='client.34343 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi005=a;smithi186=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:19:30.285 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:29 smithi005 ceph-mon[29523]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:19:30.285 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:29 smithi005 ceph-mon[29523]: Saving service node-exporter spec with placement smithi005=a;smithi186=b;count:2 2024-03-22T05:19:30.285 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:29 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:19:30.285 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:29 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:19:30.285 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:29 smithi005 ceph-mon[24793]: [22/Mar/2024:05:19:28] ENGINE Bus STARTING 2024-03-22T05:19:30.286 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:29 smithi005 ceph-mon[24793]: [22/Mar/2024:05:19:28] ENGINE Serving on https://172.21.15.5:7150 2024-03-22T05:19:30.286 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:29 smithi005 ceph-mon[24793]: [22/Mar/2024:05:19:28] ENGINE Bus STARTED 2024-03-22T05:19:30.286 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:29 smithi005 ceph-mon[24793]: mgrmap e19: y(active, since 1.9302s), standbys: x 2024-03-22T05:19:30.286 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:29 smithi005 ceph-mon[24793]: from='client.34343 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi005=a;smithi186=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:19:30.286 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:29 smithi005 ceph-mon[24793]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:19:30.286 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:29 smithi005 ceph-mon[24793]: Saving service node-exporter spec with placement smithi005=a;smithi186=b;count:2 2024-03-22T05:19:30.286 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:29 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:19:30.286 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:29 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:19:31.078 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:30 smithi186 ceph-mon[25720]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:19:31.078 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:30 smithi186 ceph-mon[25720]: mgrmap e20: y(active, since 2s), standbys: x 2024-03-22T05:19:31.078 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:30 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:19:31.078 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:30 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:19:31.078 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:30 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:19:31.078 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:30 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:19:31.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:30 smithi005 ceph-mon[29523]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:19:31.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:30 smithi005 ceph-mon[29523]: mgrmap e20: y(active, since 2s), standbys: x 2024-03-22T05:19:31.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:30 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:19:31.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:30 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:19:31.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:30 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:19:31.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:30 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:19:31.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:30 smithi005 ceph-mon[24793]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:19:31.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:30 smithi005 ceph-mon[24793]: mgrmap e20: y(active, since 2s), standbys: x 2024-03-22T05:19:31.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:30 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:19:31.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:30 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:19:31.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:30 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:19:31.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:30 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:19:32.042 INFO:teuthology.orchestra.run.smithi186.stdout:Scheduled alertmanager update... 2024-03-22T05:19:32.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:31 smithi186 ceph-mon[25720]: Updating smithi005:/etc/ceph/ceph.conf 2024-03-22T05:19:32.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:31 smithi186 ceph-mon[25720]: Updating smithi005:/etc/ceph/ceph.client.admin.keyring 2024-03-22T05:19:32.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:31 smithi186 ceph-mon[25720]: Updating smithi186:/etc/ceph/ceph.conf 2024-03-22T05:19:32.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:31 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:19:32.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:31 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:19:32.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:31 smithi005 ceph-mon[24793]: Updating smithi005:/etc/ceph/ceph.conf 2024-03-22T05:19:32.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:31 smithi005 ceph-mon[24793]: Updating smithi005:/etc/ceph/ceph.client.admin.keyring 2024-03-22T05:19:32.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:31 smithi005 ceph-mon[24793]: Updating smithi186:/etc/ceph/ceph.conf 2024-03-22T05:19:32.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:31 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:19:32.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:31 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:19:32.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:31 smithi005 ceph-mon[29523]: Updating smithi005:/etc/ceph/ceph.conf 2024-03-22T05:19:32.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:31 smithi005 ceph-mon[29523]: Updating smithi005:/etc/ceph/ceph.client.admin.keyring 2024-03-22T05:19:32.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:31 smithi005 ceph-mon[29523]: Updating smithi186:/etc/ceph/ceph.conf 2024-03-22T05:19:32.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:31 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:19:32.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:31 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:19:32.692 DEBUG:teuthology.orchestra.run.smithi005:alertmanager.a> sudo journalctl -f -n 0 -u ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@alertmanager.a.service 2024-03-22T05:19:32.695 INFO:tasks.cephadm:Adding grafana.a on smithi186 2024-03-22T05:19:32.695 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -- ceph orch apply grafana '1;smithi186=a' 2024-03-22T05:19:33.286 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:33 smithi186 ceph-mon[25720]: Updating smithi186:/etc/ceph/ceph.client.admin.keyring 2024-03-22T05:19:33.287 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:33 smithi186 ceph-mon[25720]: Deploying daemon prometheus.a on smithi186 2024-03-22T05:19:33.287 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:33 smithi186 ceph-mon[25720]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:19:33.287 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:33 smithi186 ceph-mon[25720]: from='client.34367 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi005=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:19:33.287 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:33 smithi186 ceph-mon[25720]: Saving service alertmanager spec with placement smithi005=a;count:1 2024-03-22T05:19:33.287 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:33 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:19:33.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:33 smithi005 ceph-mon[24793]: Updating smithi186:/etc/ceph/ceph.client.admin.keyring 2024-03-22T05:19:33.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:33 smithi005 ceph-mon[24793]: Deploying daemon prometheus.a on smithi186 2024-03-22T05:19:33.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:33 smithi005 ceph-mon[24793]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:19:33.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:33 smithi005 ceph-mon[24793]: from='client.34367 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi005=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:19:33.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:33 smithi005 ceph-mon[24793]: Saving service alertmanager spec with placement smithi005=a;count:1 2024-03-22T05:19:33.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:33 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:19:33.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:33 smithi005 ceph-mon[29523]: Updating smithi186:/etc/ceph/ceph.client.admin.keyring 2024-03-22T05:19:33.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:33 smithi005 ceph-mon[29523]: Deploying daemon prometheus.a on smithi186 2024-03-22T05:19:33.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:33 smithi005 ceph-mon[29523]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:19:33.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:33 smithi005 ceph-mon[29523]: from='client.34367 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi005=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:19:33.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:33 smithi005 ceph-mon[29523]: Saving service alertmanager spec with placement smithi005=a;count:1 2024-03-22T05:19:33.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:33 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:19:34.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:34 smithi005 ceph-mon[29523]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:19:34.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:34 smithi005 ceph-mon[24793]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:19:34.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:34 smithi186 ceph-mon[25720]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:19:35.711 INFO:teuthology.orchestra.run.smithi186.stdout:Scheduled grafana update... 2024-03-22T05:19:36.714 DEBUG:teuthology.orchestra.run.smithi186:grafana.a> sudo journalctl -f -n 0 -u ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@grafana.a.service 2024-03-22T05:19:36.716 INFO:tasks.cephadm:Setting up client nodes... 2024-03-22T05:19:36.719 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-22T05:19:37.036 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:36 smithi005 ceph-mon[24793]: from='client.34373 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi186=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:19:37.036 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:36 smithi005 ceph-mon[24793]: Saving service grafana spec with placement smithi186=a;count:1 2024-03-22T05:19:37.036 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:36 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:19:37.036 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:36 smithi005 ceph-mon[24793]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:19:37.036 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:36 smithi005 ceph-mon[29523]: from='client.34373 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi186=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:19:37.036 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:36 smithi005 ceph-mon[29523]: Saving service grafana spec with placement smithi186=a;count:1 2024-03-22T05:19:37.036 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:36 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:19:37.037 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:36 smithi005 ceph-mon[29523]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:19:37.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:36 smithi186 ceph-mon[25720]: from='client.34373 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi186=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:19:37.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:36 smithi186 ceph-mon[25720]: Saving service grafana spec with placement smithi186=a;count:1 2024-03-22T05:19:37.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:36 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:19:37.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:36 smithi186 ceph-mon[25720]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:19:38.746 INFO:teuthology.orchestra.run.smithi005.stdout:[client.0] 2024-03-22T05:19:38.746 INFO:teuthology.orchestra.run.smithi005.stdout: key = AQDqFP1lFYRLLBAAiVWRfnor+M7Vod8EIJYgOw== 2024-03-22T05:19:39.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:39 smithi186 ceph-mon[25720]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:19:39.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:39 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/2765214971' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-22T05:19:39.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:39 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/2765214971' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-22T05:19:39.390 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:39 smithi005 ceph-mon[24793]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:19:39.390 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:39 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/2765214971' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-22T05:19:39.390 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:39 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/2765214971' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-22T05:19:39.390 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:39 smithi005 ceph-mon[29523]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:19:39.390 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:39 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/2765214971' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-22T05:19:39.390 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:39 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/2765214971' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-22T05:19:39.415 DEBUG:teuthology.orchestra.run.smithi005:> set -ex 2024-03-22T05:19:39.415 DEBUG:teuthology.orchestra.run.smithi005:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-22T05:19:39.415 DEBUG:teuthology.orchestra.run.smithi005:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-22T05:19:39.461 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-22T05:19:41.273 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:41 smithi186 ceph-mon[25720]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:19:41.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:41 smithi005 ceph-mon[24793]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:19:41.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:41 smithi005 ceph-mon[29523]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:19:43.209 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:43 smithi186 ceph-mon[25720]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:19:43.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:43 smithi005 ceph-mon[24793]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:19:43.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:43 smithi005 ceph-mon[29523]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:19:44.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:44 smithi005 ceph-mon[24793]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:19:44.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:44 smithi005 ceph-mon[29523]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:19:44.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:44 smithi186 ceph-mon[25720]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:19:45.615 INFO:teuthology.orchestra.run.smithi186.stdout:[client.1] 2024-03-22T05:19:45.615 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQDxFP1lJmttJBAAIxGQ9IuJsFnuqASWu+buOg== 2024-03-22T05:19:45.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:45 smithi186 ceph-mon[25720]: from='client.? 172.21.15.186:0/2099600202' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-22T05:19:45.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:45 smithi186 ceph-mon[25720]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-22T05:19:45.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:45 smithi186 ceph-mon[25720]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-22T05:19:45.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:45 smithi005 ceph-mon[29523]: from='client.? 172.21.15.186:0/2099600202' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-22T05:19:45.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:45 smithi005 ceph-mon[29523]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-22T05:19:45.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:45 smithi005 ceph-mon[29523]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-22T05:19:45.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:45 smithi005 ceph-mon[24793]: from='client.? 172.21.15.186:0/2099600202' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-22T05:19:45.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:45 smithi005 ceph-mon[24793]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-22T05:19:45.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:45 smithi005 ceph-mon[24793]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-22T05:19:46.177 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-03-22T05:19:46.177 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-03-22T05:19:46.177 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-22T05:19:46.221 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-22T05:19:46.221 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-22T05:19:46.221 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -- ceph mgr dump --format=json 2024-03-22T05:19:46.432 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:19:46 smithi186 systemd[1]: Started Ceph prometheus.a for f59f36e2-e80a-11ee-95cd-87774f69a715. 2024-03-22T05:19:46.448 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T05:19:46.792 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:46 smithi005 ceph-mon[24793]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:19:46.793 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:46 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:19:46.793 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:46 smithi005 ceph-mon[29523]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:19:46.793 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:46 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:19:46.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:46 smithi186 ceph-mon[25720]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:19:46.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:46 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:19:47.785 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:47 smithi005 ceph-mon[24793]: Deploying daemon node-exporter.a on smithi005 2024-03-22T05:19:47.785 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:47 smithi005 ceph-mon[29523]: Deploying daemon node-exporter.a on smithi005 2024-03-22T05:19:48.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:47 smithi186 ceph-mon[25720]: Deploying daemon node-exporter.a on smithi005 2024-03-22T05:19:48.194 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:19:48 smithi005 systemd[1]: Starting Ceph node-exporter.a for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:19:48.329 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-22T05:19:48.452 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:19:48 smithi005 bash[44234]: Trying to pull quay.io/prometheus/node-exporter:v1.3.1... 2024-03-22T05:19:48.972 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":20,"active_gid":14433,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6800","nonce":3281603100},{"type":"v1","addr":"172.21.15.5:6801","nonce":3281603100}]},"active_addr":"172.21.15.5:6801/3281603100","active_change":"2024-03-22T05:19:27.028803+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[{"gid":34331,"name":"x","mgr_features":4540138303579357183,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.5:8443/","prometheus":"http://172.21.15.5:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":59,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.5:0","nonce":2894494732}]},{"addrvec":[{"type":"v2","addr":"172.21.15.5:0","nonce":2754168109}]},{"addrvec":[{"type":"v2","addr":"172.21.15.5:0","nonce":4022295834}]},{"addrvec":[{"type":"v2","addr":"172.21.15.5:0","nonce":3115554819}]}]}} 2024-03-22T05:19:48.978 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-22T05:19:48.979 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-22T05:19:48.979 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-22T05:19:49.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:49 smithi005 ceph-mon[24793]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:19:49.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:49 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:19:49.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:49 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/4266286450' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-22T05:19:49.203 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:19:48 smithi005 bash[44234]: Getting image source signatures 2024-03-22T05:19:49.203 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:19:48 smithi005 bash[44234]: Copying blob sha256:b5db1e299295edf3005515ab7879c1df64a33c185d3a7a23aa4dcaa17d26f7b3 2024-03-22T05:19:49.203 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:19:48 smithi005 bash[44234]: Copying blob sha256:b45d31ee2d7f9f452678a85b0c837c29e12089f31ee8dbac6c8c24dfa4054a30 2024-03-22T05:19:49.203 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:19:48 smithi005 bash[44234]: Copying blob sha256:aa2a8d90b84cb2a9c422e7005cd166a008ccf22ef5d7d4f07128478585ce35ea 2024-03-22T05:19:49.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:49 smithi005 ceph-mon[29523]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:19:49.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:49 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:19:49.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:49 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/4266286450' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-22T05:19:49.214 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T05:19:49.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:49 smithi186 ceph-mon[25720]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:19:49.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:49 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:19:49.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:49 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/4266286450' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-22T05:19:50.349 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:19:50 smithi005 bash[44234]: Copying config sha256:1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 2024-03-22T05:19:50.920 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:19:50 smithi005 bash[44234]: Writing manifest to image destination 2024-03-22T05:19:51.194 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:51 smithi005 ceph-mon[29523]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:19:51.195 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:51 smithi005 ceph-mon[24793]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:19:51.195 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:19:50 smithi005 podman[44234]: 2024-03-22 05:19:50.916249738 +0000 UTC m=+2.620868989 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-03-22T05:19:51.195 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:19:51 smithi005 podman[44234]: 2024-03-22 05:19:51.114471667 +0000 UTC m=+2.819090889 container create 1ee16c40186bbe0f00ec98516f41d9f19078571cbf8f93f9ab51c9f0103b574e (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-22T05:19:51.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:51 smithi186 ceph-mon[25720]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:19:51.910 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:19:51 smithi005 podman[44234]: 2024-03-22 05:19:51.760616666 +0000 UTC m=+3.465235897 container init 1ee16c40186bbe0f00ec98516f41d9f19078571cbf8f93f9ab51c9f0103b574e (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-22T05:19:51.910 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:19:51 smithi005 podman[44234]: 2024-03-22 05:19:51.7633616 +0000 UTC m=+3.467980816 container start 1ee16c40186bbe0f00ec98516f41d9f19078571cbf8f93f9ab51c9f0103b574e (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-22T05:19:51.910 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:19:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[44392]: ts=2024-03-22T05:19:51.767Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-03-22T05:19:51.910 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:19:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[44392]: ts=2024-03-22T05:19:51.767Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-03-22T05:19:51.910 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:19:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[44392]: ts=2024-03-22T05:19:51.768Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-03-22T05:19:51.910 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:19:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[44392]: ts=2024-03-22T05:19:51.768Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-22T05:19:51.910 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:19:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[44392]: ts=2024-03-22T05:19:51.769Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-03-22T05:19:51.910 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:19:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[44392]: ts=2024-03-22T05:19:51.769Z caller=node_exporter.go:115 level=info collector=arp 2024-03-22T05:19:51.911 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:19:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[44392]: ts=2024-03-22T05:19:51.769Z caller=node_exporter.go:115 level=info collector=bcache 2024-03-22T05:19:51.911 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:19:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[44392]: ts=2024-03-22T05:19:51.769Z caller=node_exporter.go:115 level=info collector=bonding 2024-03-22T05:19:51.911 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:19:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[44392]: ts=2024-03-22T05:19:51.769Z caller=node_exporter.go:115 level=info collector=btrfs 2024-03-22T05:19:51.911 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:19:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[44392]: ts=2024-03-22T05:19:51.769Z caller=node_exporter.go:115 level=info collector=conntrack 2024-03-22T05:19:51.911 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:19:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[44392]: ts=2024-03-22T05:19:51.769Z caller=node_exporter.go:115 level=info collector=cpu 2024-03-22T05:19:51.911 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:19:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[44392]: ts=2024-03-22T05:19:51.769Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-03-22T05:19:51.911 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:19:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[44392]: ts=2024-03-22T05:19:51.769Z caller=node_exporter.go:115 level=info collector=diskstats 2024-03-22T05:19:51.911 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:19:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[44392]: ts=2024-03-22T05:19:51.769Z caller=node_exporter.go:115 level=info collector=dmi 2024-03-22T05:19:51.911 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:19:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[44392]: ts=2024-03-22T05:19:51.769Z caller=node_exporter.go:115 level=info collector=edac 2024-03-22T05:19:51.911 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:19:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[44392]: ts=2024-03-22T05:19:51.769Z caller=node_exporter.go:115 level=info collector=entropy 2024-03-22T05:19:51.911 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:19:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[44392]: ts=2024-03-22T05:19:51.769Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-03-22T05:19:51.912 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:19:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[44392]: ts=2024-03-22T05:19:51.769Z caller=node_exporter.go:115 level=info collector=filefd 2024-03-22T05:19:51.912 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:19:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[44392]: ts=2024-03-22T05:19:51.769Z caller=node_exporter.go:115 level=info collector=filesystem 2024-03-22T05:19:51.912 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:19:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[44392]: ts=2024-03-22T05:19:51.769Z caller=node_exporter.go:115 level=info collector=hwmon 2024-03-22T05:19:51.912 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:19:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[44392]: ts=2024-03-22T05:19:51.769Z caller=node_exporter.go:115 level=info collector=infiniband 2024-03-22T05:19:51.912 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:19:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[44392]: ts=2024-03-22T05:19:51.769Z caller=node_exporter.go:115 level=info collector=ipvs 2024-03-22T05:19:51.912 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:19:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[44392]: ts=2024-03-22T05:19:51.769Z caller=node_exporter.go:115 level=info collector=loadavg 2024-03-22T05:19:51.912 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:19:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[44392]: ts=2024-03-22T05:19:51.769Z caller=node_exporter.go:115 level=info collector=mdadm 2024-03-22T05:19:51.912 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:19:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[44392]: ts=2024-03-22T05:19:51.769Z caller=node_exporter.go:115 level=info collector=meminfo 2024-03-22T05:19:51.912 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:19:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[44392]: ts=2024-03-22T05:19:51.769Z caller=node_exporter.go:115 level=info collector=netclass 2024-03-22T05:19:51.912 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:19:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[44392]: ts=2024-03-22T05:19:51.769Z caller=node_exporter.go:115 level=info collector=netdev 2024-03-22T05:19:51.912 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:19:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[44392]: ts=2024-03-22T05:19:51.769Z caller=node_exporter.go:115 level=info collector=netstat 2024-03-22T05:19:51.912 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:19:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[44392]: ts=2024-03-22T05:19:51.769Z caller=node_exporter.go:115 level=info collector=nfs 2024-03-22T05:19:51.913 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:19:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[44392]: ts=2024-03-22T05:19:51.769Z caller=node_exporter.go:115 level=info collector=nfsd 2024-03-22T05:19:51.913 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:19:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[44392]: ts=2024-03-22T05:19:51.769Z caller=node_exporter.go:115 level=info collector=nvme 2024-03-22T05:19:51.913 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:19:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[44392]: ts=2024-03-22T05:19:51.769Z caller=node_exporter.go:115 level=info collector=os 2024-03-22T05:19:51.913 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:19:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[44392]: ts=2024-03-22T05:19:51.769Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-03-22T05:19:51.913 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:19:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[44392]: ts=2024-03-22T05:19:51.769Z caller=node_exporter.go:115 level=info collector=pressure 2024-03-22T05:19:51.913 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:19:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[44392]: ts=2024-03-22T05:19:51.769Z caller=node_exporter.go:115 level=info collector=rapl 2024-03-22T05:19:51.913 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:19:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[44392]: ts=2024-03-22T05:19:51.769Z caller=node_exporter.go:115 level=info collector=schedstat 2024-03-22T05:19:51.913 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:19:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[44392]: ts=2024-03-22T05:19:51.769Z caller=node_exporter.go:115 level=info collector=sockstat 2024-03-22T05:19:51.913 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:19:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[44392]: ts=2024-03-22T05:19:51.769Z caller=node_exporter.go:115 level=info collector=softnet 2024-03-22T05:19:51.913 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:19:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[44392]: ts=2024-03-22T05:19:51.769Z caller=node_exporter.go:115 level=info collector=stat 2024-03-22T05:19:51.913 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:19:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[44392]: ts=2024-03-22T05:19:51.769Z caller=node_exporter.go:115 level=info collector=tapestats 2024-03-22T05:19:51.913 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:19:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[44392]: ts=2024-03-22T05:19:51.769Z caller=node_exporter.go:115 level=info collector=textfile 2024-03-22T05:19:51.913 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:19:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[44392]: ts=2024-03-22T05:19:51.769Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-03-22T05:19:51.914 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:19:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[44392]: ts=2024-03-22T05:19:51.769Z caller=node_exporter.go:115 level=info collector=time 2024-03-22T05:19:51.914 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:19:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[44392]: ts=2024-03-22T05:19:51.769Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-03-22T05:19:51.914 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:19:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[44392]: ts=2024-03-22T05:19:51.769Z caller=node_exporter.go:115 level=info collector=uname 2024-03-22T05:19:51.914 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:19:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[44392]: ts=2024-03-22T05:19:51.769Z caller=node_exporter.go:115 level=info collector=vmstat 2024-03-22T05:19:51.914 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:19:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[44392]: ts=2024-03-22T05:19:51.769Z caller=node_exporter.go:115 level=info collector=xfs 2024-03-22T05:19:51.914 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:19:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[44392]: ts=2024-03-22T05:19:51.769Z caller=node_exporter.go:115 level=info collector=zfs 2024-03-22T05:19:51.914 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:19:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[44392]: ts=2024-03-22T05:19:51.769Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-03-22T05:19:51.914 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:19:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[44392]: ts=2024-03-22T05:19:51.769Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-03-22T05:19:51.914 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:19:51 smithi005 bash[44234]: 1ee16c40186bbe0f00ec98516f41d9f19078571cbf8f93f9ab51c9f0103b574e 2024-03-22T05:19:51.914 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:19:51 smithi005 systemd[1]: Started Ceph node-exporter.a for f59f36e2-e80a-11ee-95cd-87774f69a715. 2024-03-22T05:19:52.470 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-22T05:19:52.470 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":59,"fsid":"f59f36e2-e80a-11ee-95cd-87774f69a715","created":"2024-03-22T05:14:18.934387+0000","modified":"2024-03-22T05:19:27.028462+0000","last_up_change":"2024-03-22T05:19:11.125488+0000","last_in_change":"2024-03-22T05:18:53.961201+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-22T05:17:17.465016+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"a0063741-df11-4674-9d0e-0ac5161bd1b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":2350146241},{"type":"v1","addr":"172.21.15.5:6803","nonce":2350146241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":2350146241},{"type":"v1","addr":"172.21.15.5:6805","nonce":2350146241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":2350146241},{"type":"v1","addr":"172.21.15.5:6809","nonce":2350146241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":2350146241},{"type":"v1","addr":"172.21.15.5:6807","nonce":2350146241}]},"public_addr":"172.21.15.5:6803/2350146241","cluster_addr":"172.21.15.5:6805/2350146241","heartbeat_back_addr":"172.21.15.5:6809/2350146241","heartbeat_front_addr":"172.21.15.5:6807/2350146241","state":["exists","up"]},{"osd":1,"uuid":"ef4e89fc-61a9-46c7-8357-2378e58055b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6810","nonce":790214800},{"type":"v1","addr":"172.21.15.5:6811","nonce":790214800}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":790214800},{"type":"v1","addr":"172.21.15.5:6813","nonce":790214800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6816","nonce":790214800},{"type":"v1","addr":"172.21.15.5:6817","nonce":790214800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":790214800},{"type":"v1","addr":"172.21.15.5:6815","nonce":790214800}]},"public_addr":"172.21.15.5:6811/790214800","cluster_addr":"172.21.15.5:6813/790214800","heartbeat_back_addr":"172.21.15.5:6817/790214800","heartbeat_front_addr":"172.21.15.5:6815/790214800","state":["exists","up"]},{"osd":2,"uuid":"c1365858-72bc-4a3a-84cb-ac8f1beaa725","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":3354758722},{"type":"v1","addr":"172.21.15.5:6819","nonce":3354758722}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":3354758722},{"type":"v1","addr":"172.21.15.5:6821","nonce":3354758722}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":3354758722},{"type":"v1","addr":"172.21.15.5:6825","nonce":3354758722}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6822","nonce":3354758722},{"type":"v1","addr":"172.21.15.5:6823","nonce":3354758722}]},"public_addr":"172.21.15.5:6819/3354758722","cluster_addr":"172.21.15.5:6821/3354758722","heartbeat_back_addr":"172.21.15.5:6825/3354758722","heartbeat_front_addr":"172.21.15.5:6823/3354758722","state":["exists","up"]},{"osd":3,"uuid":"ae0fa9b1-b54d-4c9a-b5d0-33ae91dcf98d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":3829723611},{"type":"v1","addr":"172.21.15.5:6827","nonce":3829723611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6828","nonce":3829723611},{"type":"v1","addr":"172.21.15.5:6829","nonce":3829723611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":3829723611},{"type":"v1","addr":"172.21.15.5:6833","nonce":3829723611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":3829723611},{"type":"v1","addr":"172.21.15.5:6831","nonce":3829723611}]},"public_addr":"172.21.15.5:6827/3829723611","cluster_addr":"172.21.15.5:6829/3829723611","heartbeat_back_addr":"172.21.15.5:6833/3829723611","heartbeat_front_addr":"172.21.15.5:6831/3829723611","state":["exists","up"]},{"osd":4,"uuid":"7313be61-8567-42c4-93a0-8ff3f2e30da3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":2861669931},{"type":"v1","addr":"172.21.15.186:6801","nonce":2861669931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":2861669931},{"type":"v1","addr":"172.21.15.186:6803","nonce":2861669931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":2861669931},{"type":"v1","addr":"172.21.15.186:6807","nonce":2861669931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":2861669931},{"type":"v1","addr":"172.21.15.186:6805","nonce":2861669931}]},"public_addr":"172.21.15.186:6801/2861669931","cluster_addr":"172.21.15.186:6803/2861669931","heartbeat_back_addr":"172.21.15.186:6807/2861669931","heartbeat_front_addr":"172.21.15.186:6805/2861669931","state":["exists","up"]},{"osd":5,"uuid":"edeed5ee-7262-4ce5-a32f-cef71b2d6acf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":554926692},{"type":"v1","addr":"172.21.15.186:6809","nonce":554926692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":554926692},{"type":"v1","addr":"172.21.15.186:6811","nonce":554926692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":554926692},{"type":"v1","addr":"172.21.15.186:6815","nonce":554926692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":554926692},{"type":"v1","addr":"172.21.15.186:6813","nonce":554926692}]},"public_addr":"172.21.15.186:6809/554926692","cluster_addr":"172.21.15.186:6811/554926692","heartbeat_back_addr":"172.21.15.186:6815/554926692","heartbeat_front_addr":"172.21.15.186:6813/554926692","state":["exists","up"]},{"osd":6,"uuid":"b979e26c-6b94-434f-8c95-9b9a8713eb78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":2574435174},{"type":"v1","addr":"172.21.15.186:6817","nonce":2574435174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":2574435174},{"type":"v1","addr":"172.21.15.186:6819","nonce":2574435174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":2574435174},{"type":"v1","addr":"172.21.15.186:6823","nonce":2574435174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":2574435174},{"type":"v1","addr":"172.21.15.186:6821","nonce":2574435174}]},"public_addr":"172.21.15.186:6817/2574435174","cluster_addr":"172.21.15.186:6819/2574435174","heartbeat_back_addr":"172.21.15.186:6823/2574435174","heartbeat_front_addr":"172.21.15.186:6821/2574435174","state":["exists","up"]},{"osd":7,"uuid":"0ae28e60-285b-423e-b686-f536cffbd595","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3520054895},{"type":"v1","addr":"172.21.15.186:6825","nonce":3520054895}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3520054895},{"type":"v1","addr":"172.21.15.186:6827","nonce":3520054895}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3520054895},{"type":"v1","addr":"172.21.15.186:6831","nonce":3520054895}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3520054895},{"type":"v1","addr":"172.21.15.186:6829","nonce":3520054895}]},"public_addr":"172.21.15.186:6825/3520054895","cluster_addr":"172.21.15.186:6827/3520054895","heartbeat_back_addr":"172.21.15.186:6831/3520054895","heartbeat_front_addr":"172.21.15.186:6829/3520054895","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T05:16:28.343379+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T05:16:51.136049+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T05:17:15.010413+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T05:17:38.810252+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T05:17:59.769820+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T05:18:21.904758+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T05:18:45.000458+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T05:19:09.910476+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.5:0/757446544":"2024-03-23T05:19:27.028432+0000","172.21.15.5:6800/294522234":"2024-03-23T05:19:27.028432+0000","172.21.15.5:0/2650789828":"2024-03-23T05:15:09.085731+0000","172.21.15.5:6801/2160302520":"2024-03-23T05:15:09.085731+0000","172.21.15.5:0/1122231072":"2024-03-23T05:19:27.028432+0000","172.21.15.5:6800/2160302520":"2024-03-23T05:15:09.085731+0000","172.21.15.5:0/2169353506":"2024-03-23T05:15:09.085731+0000","172.21.15.5:0/1346312444":"2024-03-23T05:19:27.028432+0000","172.21.15.5:6801/294522234":"2024-03-23T05:19:27.028432+0000","172.21.15.5:0/3301532190":"2024-03-23T05:14:47.571432+0000","172.21.15.5:0/2858060116":"2024-03-23T05:14:47.571432+0000","172.21.15.5:6800/392664475":"2024-03-23T05:14:47.571432+0000","172.21.15.5:6801/392664475":"2024-03-23T05:14:47.571432+0000","172.21.15.5:0/1891765076":"2024-03-23T05:19:27.028432+0000","172.21.15.5:0/2331329527":"2024-03-23T05:15:09.085731+0000","172.21.15.5:0/789815545":"2024-03-23T05:14:47.571432+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-22T05:19:52.702 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:19:52 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:19:52] "GET /metrics HTTP/1.1" 200 191466 "" "Prometheus/2.33.4" 2024-03-22T05:19:52.920 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:19:52 smithi186 bash[41044]: Trying to pull quay.io/prometheus/node-exporter:v1.3.1... 2024-03-22T05:19:53.159 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:52 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:19:53.159 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:52 smithi005 ceph-mon[29523]: Deploying daemon node-exporter.b on smithi186 2024-03-22T05:19:53.159 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:52 smithi005 ceph-mon[29523]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:19:53.159 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:52 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/1977150831' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T05:19:53.159 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:52 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:19:53.159 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:52 smithi005 ceph-mon[24793]: Deploying daemon node-exporter.b on smithi186 2024-03-22T05:19:53.159 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:52 smithi005 ceph-mon[24793]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:19:53.159 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:52 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/1977150831' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T05:19:53.184 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-22T05:19:53.184 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-22T05:19:53.192 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:52 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:19:53.192 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:52 smithi186 ceph-mon[25720]: Deploying daemon node-exporter.b on smithi186 2024-03-22T05:19:53.193 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:52 smithi186 ceph-mon[25720]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:19:53.193 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:52 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/1977150831' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T05:19:53.193 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:19:53 smithi186 bash[41044]: Getting image source signatures 2024-03-22T05:19:53.193 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:19:53 smithi186 bash[41044]: Copying blob sha256:b5db1e299295edf3005515ab7879c1df64a33c185d3a7a23aa4dcaa17d26f7b3 2024-03-22T05:19:53.193 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:19:53 smithi186 bash[41044]: Copying blob sha256:aa2a8d90b84cb2a9c422e7005cd166a008ccf22ef5d7d4f07128478585ce35ea 2024-03-22T05:19:53.193 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:19:53 smithi186 bash[41044]: Copying blob sha256:b45d31ee2d7f9f452678a85b0c837c29e12089f31ee8dbac6c8c24dfa4054a30 2024-03-22T05:19:53.405 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T05:19:53.544 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:19:53 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:19:53] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-22T05:19:54.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:54 smithi005 ceph-mon[29523]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:19:54.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:54 smithi005 ceph-mon[24793]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:19:54.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:54 smithi186 ceph-mon[25720]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:19:55.043 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:19:54 smithi186 bash[41044]: Copying config sha256:1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 2024-03-22T05:19:55.043 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:19:54 smithi186 bash[41044]: Writing manifest to image destination 2024-03-22T05:19:55.251 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-22T05:19:55.251 INFO:teuthology.orchestra.run.smithi005.stdout:{"epoch":59,"fsid":"f59f36e2-e80a-11ee-95cd-87774f69a715","created":"2024-03-22T05:14:18.934387+0000","modified":"2024-03-22T05:19:27.028462+0000","last_up_change":"2024-03-22T05:19:11.125488+0000","last_in_change":"2024-03-22T05:18:53.961201+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-22T05:17:17.465016+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"a0063741-df11-4674-9d0e-0ac5161bd1b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6802","nonce":2350146241},{"type":"v1","addr":"172.21.15.5:6803","nonce":2350146241}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6804","nonce":2350146241},{"type":"v1","addr":"172.21.15.5:6805","nonce":2350146241}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6808","nonce":2350146241},{"type":"v1","addr":"172.21.15.5:6809","nonce":2350146241}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6806","nonce":2350146241},{"type":"v1","addr":"172.21.15.5:6807","nonce":2350146241}]},"public_addr":"172.21.15.5:6803/2350146241","cluster_addr":"172.21.15.5:6805/2350146241","heartbeat_back_addr":"172.21.15.5:6809/2350146241","heartbeat_front_addr":"172.21.15.5:6807/2350146241","state":["exists","up"]},{"osd":1,"uuid":"ef4e89fc-61a9-46c7-8357-2378e58055b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6810","nonce":790214800},{"type":"v1","addr":"172.21.15.5:6811","nonce":790214800}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6812","nonce":790214800},{"type":"v1","addr":"172.21.15.5:6813","nonce":790214800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6816","nonce":790214800},{"type":"v1","addr":"172.21.15.5:6817","nonce":790214800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6814","nonce":790214800},{"type":"v1","addr":"172.21.15.5:6815","nonce":790214800}]},"public_addr":"172.21.15.5:6811/790214800","cluster_addr":"172.21.15.5:6813/790214800","heartbeat_back_addr":"172.21.15.5:6817/790214800","heartbeat_front_addr":"172.21.15.5:6815/790214800","state":["exists","up"]},{"osd":2,"uuid":"c1365858-72bc-4a3a-84cb-ac8f1beaa725","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6818","nonce":3354758722},{"type":"v1","addr":"172.21.15.5:6819","nonce":3354758722}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6820","nonce":3354758722},{"type":"v1","addr":"172.21.15.5:6821","nonce":3354758722}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6824","nonce":3354758722},{"type":"v1","addr":"172.21.15.5:6825","nonce":3354758722}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6822","nonce":3354758722},{"type":"v1","addr":"172.21.15.5:6823","nonce":3354758722}]},"public_addr":"172.21.15.5:6819/3354758722","cluster_addr":"172.21.15.5:6821/3354758722","heartbeat_back_addr":"172.21.15.5:6825/3354758722","heartbeat_front_addr":"172.21.15.5:6823/3354758722","state":["exists","up"]},{"osd":3,"uuid":"ae0fa9b1-b54d-4c9a-b5d0-33ae91dcf98d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6826","nonce":3829723611},{"type":"v1","addr":"172.21.15.5:6827","nonce":3829723611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6828","nonce":3829723611},{"type":"v1","addr":"172.21.15.5:6829","nonce":3829723611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6832","nonce":3829723611},{"type":"v1","addr":"172.21.15.5:6833","nonce":3829723611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.5:6830","nonce":3829723611},{"type":"v1","addr":"172.21.15.5:6831","nonce":3829723611}]},"public_addr":"172.21.15.5:6827/3829723611","cluster_addr":"172.21.15.5:6829/3829723611","heartbeat_back_addr":"172.21.15.5:6833/3829723611","heartbeat_front_addr":"172.21.15.5:6831/3829723611","state":["exists","up"]},{"osd":4,"uuid":"7313be61-8567-42c4-93a0-8ff3f2e30da3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":2861669931},{"type":"v1","addr":"172.21.15.186:6801","nonce":2861669931}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":2861669931},{"type":"v1","addr":"172.21.15.186:6803","nonce":2861669931}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":2861669931},{"type":"v1","addr":"172.21.15.186:6807","nonce":2861669931}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":2861669931},{"type":"v1","addr":"172.21.15.186:6805","nonce":2861669931}]},"public_addr":"172.21.15.186:6801/2861669931","cluster_addr":"172.21.15.186:6803/2861669931","heartbeat_back_addr":"172.21.15.186:6807/2861669931","heartbeat_front_addr":"172.21.15.186:6805/2861669931","state":["exists","up"]},{"osd":5,"uuid":"edeed5ee-7262-4ce5-a32f-cef71b2d6acf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":554926692},{"type":"v1","addr":"172.21.15.186:6809","nonce":554926692}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":554926692},{"type":"v1","addr":"172.21.15.186:6811","nonce":554926692}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":554926692},{"type":"v1","addr":"172.21.15.186:6815","nonce":554926692}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":554926692},{"type":"v1","addr":"172.21.15.186:6813","nonce":554926692}]},"public_addr":"172.21.15.186:6809/554926692","cluster_addr":"172.21.15.186:6811/554926692","heartbeat_back_addr":"172.21.15.186:6815/554926692","heartbeat_front_addr":"172.21.15.186:6813/554926692","state":["exists","up"]},{"osd":6,"uuid":"b979e26c-6b94-434f-8c95-9b9a8713eb78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":2574435174},{"type":"v1","addr":"172.21.15.186:6817","nonce":2574435174}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":2574435174},{"type":"v1","addr":"172.21.15.186:6819","nonce":2574435174}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":2574435174},{"type":"v1","addr":"172.21.15.186:6823","nonce":2574435174}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":2574435174},{"type":"v1","addr":"172.21.15.186:6821","nonce":2574435174}]},"public_addr":"172.21.15.186:6817/2574435174","cluster_addr":"172.21.15.186:6819/2574435174","heartbeat_back_addr":"172.21.15.186:6823/2574435174","heartbeat_front_addr":"172.21.15.186:6821/2574435174","state":["exists","up"]},{"osd":7,"uuid":"0ae28e60-285b-423e-b686-f536cffbd595","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3520054895},{"type":"v1","addr":"172.21.15.186:6825","nonce":3520054895}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3520054895},{"type":"v1","addr":"172.21.15.186:6827","nonce":3520054895}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3520054895},{"type":"v1","addr":"172.21.15.186:6831","nonce":3520054895}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3520054895},{"type":"v1","addr":"172.21.15.186:6829","nonce":3520054895}]},"public_addr":"172.21.15.186:6825/3520054895","cluster_addr":"172.21.15.186:6827/3520054895","heartbeat_back_addr":"172.21.15.186:6831/3520054895","heartbeat_front_addr":"172.21.15.186:6829/3520054895","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T05:16:28.343379+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T05:16:51.136049+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T05:17:15.010413+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T05:17:38.810252+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T05:17:59.769820+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T05:18:21.904758+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T05:18:45.000458+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-22T05:19:09.910476+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.5:0/757446544":"2024-03-23T05:19:27.028432+0000","172.21.15.5:6800/294522234":"2024-03-23T05:19:27.028432+0000","172.21.15.5:0/2650789828":"2024-03-23T05:15:09.085731+0000","172.21.15.5:6801/2160302520":"2024-03-23T05:15:09.085731+0000","172.21.15.5:0/1122231072":"2024-03-23T05:19:27.028432+0000","172.21.15.5:6800/2160302520":"2024-03-23T05:15:09.085731+0000","172.21.15.5:0/2169353506":"2024-03-23T05:15:09.085731+0000","172.21.15.5:0/1346312444":"2024-03-23T05:19:27.028432+0000","172.21.15.5:6801/294522234":"2024-03-23T05:19:27.028432+0000","172.21.15.5:0/3301532190":"2024-03-23T05:14:47.571432+0000","172.21.15.5:0/2858060116":"2024-03-23T05:14:47.571432+0000","172.21.15.5:6800/392664475":"2024-03-23T05:14:47.571432+0000","172.21.15.5:6801/392664475":"2024-03-23T05:14:47.571432+0000","172.21.15.5:0/1891765076":"2024-03-23T05:19:27.028432+0000","172.21.15.5:0/2331329527":"2024-03-23T05:15:09.085731+0000","172.21.15.5:0/789815545":"2024-03-23T05:14:47.571432+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-22T05:19:55.408 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:55 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/3674237576' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T05:19:55.408 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:19:55 smithi186 podman[41044]: 2024-03-22 05:19:55.083292942 +0000 UTC m=+2.399486761 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-03-22T05:19:55.408 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:19:55 smithi186 podman[41044]: 2024-03-22 05:19:55.181809314 +0000 UTC m=+2.498003137 container create e1c552dad9306f79982a50546495ed3b583865bb1383eee35b99463ed491349d (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-22T05:19:55.408 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:19:55 smithi186 podman[41044]: 2024-03-22 05:19:55.408741162 +0000 UTC m=+2.724934994 container init e1c552dad9306f79982a50546495ed3b583865bb1383eee35b99463ed491349d (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-22T05:19:55.558 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:55 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/3674237576' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T05:19:55.559 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:55 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/3674237576' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-22T05:19:55.794 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:19:55 smithi186 podman[41044]: 2024-03-22 05:19:55.411921741 +0000 UTC m=+2.728115570 container start e1c552dad9306f79982a50546495ed3b583865bb1383eee35b99463ed491349d (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-22T05:19:55.794 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:19:55 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[41096]: ts=2024-03-22T05:19:55.415Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-03-22T05:19:55.795 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:19:55 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[41096]: ts=2024-03-22T05:19:55.415Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-03-22T05:19:55.795 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:19:55 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[41096]: ts=2024-03-22T05:19:55.416Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-03-22T05:19:55.795 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:19:55 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[41096]: ts=2024-03-22T05:19:55.416Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-22T05:19:55.795 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:19:55 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[41096]: ts=2024-03-22T05:19:55.417Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-03-22T05:19:55.795 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:19:55 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[41096]: ts=2024-03-22T05:19:55.417Z caller=node_exporter.go:115 level=info collector=arp 2024-03-22T05:19:55.795 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:19:55 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[41096]: ts=2024-03-22T05:19:55.417Z caller=node_exporter.go:115 level=info collector=bcache 2024-03-22T05:19:55.795 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:19:55 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[41096]: ts=2024-03-22T05:19:55.417Z caller=node_exporter.go:115 level=info collector=bonding 2024-03-22T05:19:55.795 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:19:55 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[41096]: ts=2024-03-22T05:19:55.417Z caller=node_exporter.go:115 level=info collector=btrfs 2024-03-22T05:19:55.795 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:19:55 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[41096]: ts=2024-03-22T05:19:55.417Z caller=node_exporter.go:115 level=info collector=conntrack 2024-03-22T05:19:55.795 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:19:55 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[41096]: ts=2024-03-22T05:19:55.417Z caller=node_exporter.go:115 level=info collector=cpu 2024-03-22T05:19:55.795 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:19:55 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[41096]: ts=2024-03-22T05:19:55.417Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-03-22T05:19:55.795 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:19:55 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[41096]: ts=2024-03-22T05:19:55.417Z caller=node_exporter.go:115 level=info collector=diskstats 2024-03-22T05:19:55.795 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:19:55 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[41096]: ts=2024-03-22T05:19:55.417Z caller=node_exporter.go:115 level=info collector=dmi 2024-03-22T05:19:55.795 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:19:55 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[41096]: ts=2024-03-22T05:19:55.417Z caller=node_exporter.go:115 level=info collector=edac 2024-03-22T05:19:55.796 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:19:55 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[41096]: ts=2024-03-22T05:19:55.417Z caller=node_exporter.go:115 level=info collector=entropy 2024-03-22T05:19:55.796 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:19:55 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[41096]: ts=2024-03-22T05:19:55.417Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-03-22T05:19:55.796 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:19:55 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[41096]: ts=2024-03-22T05:19:55.417Z caller=node_exporter.go:115 level=info collector=filefd 2024-03-22T05:19:55.796 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:19:55 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[41096]: ts=2024-03-22T05:19:55.417Z caller=node_exporter.go:115 level=info collector=filesystem 2024-03-22T05:19:55.796 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:19:55 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[41096]: ts=2024-03-22T05:19:55.417Z caller=node_exporter.go:115 level=info collector=hwmon 2024-03-22T05:19:55.796 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:19:55 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[41096]: ts=2024-03-22T05:19:55.417Z caller=node_exporter.go:115 level=info collector=infiniband 2024-03-22T05:19:55.796 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:19:55 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[41096]: ts=2024-03-22T05:19:55.417Z caller=node_exporter.go:115 level=info collector=ipvs 2024-03-22T05:19:55.796 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:19:55 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[41096]: ts=2024-03-22T05:19:55.417Z caller=node_exporter.go:115 level=info collector=loadavg 2024-03-22T05:19:55.797 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:19:55 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[41096]: ts=2024-03-22T05:19:55.417Z caller=node_exporter.go:115 level=info collector=mdadm 2024-03-22T05:19:55.797 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:19:55 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[41096]: ts=2024-03-22T05:19:55.417Z caller=node_exporter.go:115 level=info collector=meminfo 2024-03-22T05:19:55.797 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:19:55 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[41096]: ts=2024-03-22T05:19:55.417Z caller=node_exporter.go:115 level=info collector=netclass 2024-03-22T05:19:55.797 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:19:55 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[41096]: ts=2024-03-22T05:19:55.417Z caller=node_exporter.go:115 level=info collector=netdev 2024-03-22T05:19:55.797 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:19:55 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[41096]: ts=2024-03-22T05:19:55.417Z caller=node_exporter.go:115 level=info collector=netstat 2024-03-22T05:19:55.797 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:19:55 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[41096]: ts=2024-03-22T05:19:55.417Z caller=node_exporter.go:115 level=info collector=nfs 2024-03-22T05:19:55.797 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:19:55 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[41096]: ts=2024-03-22T05:19:55.417Z caller=node_exporter.go:115 level=info collector=nfsd 2024-03-22T05:19:55.797 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:19:55 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[41096]: ts=2024-03-22T05:19:55.417Z caller=node_exporter.go:115 level=info collector=nvme 2024-03-22T05:19:55.797 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:19:55 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[41096]: ts=2024-03-22T05:19:55.417Z caller=node_exporter.go:115 level=info collector=os 2024-03-22T05:19:55.797 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:19:55 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[41096]: ts=2024-03-22T05:19:55.417Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-03-22T05:19:55.797 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:19:55 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[41096]: ts=2024-03-22T05:19:55.417Z caller=node_exporter.go:115 level=info collector=pressure 2024-03-22T05:19:55.797 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:19:55 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[41096]: ts=2024-03-22T05:19:55.417Z caller=node_exporter.go:115 level=info collector=rapl 2024-03-22T05:19:55.797 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:19:55 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[41096]: ts=2024-03-22T05:19:55.417Z caller=node_exporter.go:115 level=info collector=schedstat 2024-03-22T05:19:55.797 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:19:55 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[41096]: ts=2024-03-22T05:19:55.417Z caller=node_exporter.go:115 level=info collector=sockstat 2024-03-22T05:19:55.797 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:19:55 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[41096]: ts=2024-03-22T05:19:55.417Z caller=node_exporter.go:115 level=info collector=softnet 2024-03-22T05:19:55.798 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:19:55 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[41096]: ts=2024-03-22T05:19:55.417Z caller=node_exporter.go:115 level=info collector=stat 2024-03-22T05:19:55.798 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:19:55 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[41096]: ts=2024-03-22T05:19:55.417Z caller=node_exporter.go:115 level=info collector=tapestats 2024-03-22T05:19:55.798 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:19:55 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[41096]: ts=2024-03-22T05:19:55.417Z caller=node_exporter.go:115 level=info collector=textfile 2024-03-22T05:19:55.798 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:19:55 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[41096]: ts=2024-03-22T05:19:55.417Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-03-22T05:19:55.798 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:19:55 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[41096]: ts=2024-03-22T05:19:55.417Z caller=node_exporter.go:115 level=info collector=time 2024-03-22T05:19:55.798 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:19:55 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[41096]: ts=2024-03-22T05:19:55.417Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-03-22T05:19:55.798 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:19:55 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[41096]: ts=2024-03-22T05:19:55.417Z caller=node_exporter.go:115 level=info collector=uname 2024-03-22T05:19:55.798 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:19:55 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[41096]: ts=2024-03-22T05:19:55.417Z caller=node_exporter.go:115 level=info collector=vmstat 2024-03-22T05:19:55.798 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:19:55 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[41096]: ts=2024-03-22T05:19:55.417Z caller=node_exporter.go:115 level=info collector=xfs 2024-03-22T05:19:55.798 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:19:55 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[41096]: ts=2024-03-22T05:19:55.417Z caller=node_exporter.go:115 level=info collector=zfs 2024-03-22T05:19:55.798 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:19:55 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[41096]: ts=2024-03-22T05:19:55.417Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-03-22T05:19:55.798 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:19:55 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[41096]: ts=2024-03-22T05:19:55.417Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-03-22T05:19:55.798 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:19:55 smithi186 bash[41044]: e1c552dad9306f79982a50546495ed3b583865bb1383eee35b99463ed491349d 2024-03-22T05:19:55.798 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:19:55 smithi186 systemd[1]: Started Ceph node-exporter.b for f59f36e2-e80a-11ee-95cd-87774f69a715. 2024-03-22T05:19:55.940 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-22T05:19:55.940 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-22T05:19:55.941 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-22T05:19:55.941 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-22T05:19:55.941 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-22T05:19:55.942 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-22T05:19:55.942 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-22T05:19:55.942 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-22T05:19:56.378 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T05:19:56.383 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T05:19:56.634 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T05:19:56.634 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T05:19:56.638 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T05:19:56.638 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T05:19:56.639 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T05:19:56.642 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T05:19:56.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:56 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:19:56.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:56 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:19:56.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:56 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:19:56.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:56 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:19:56.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:56 smithi005 ceph-mon[24793]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:19:56.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:56 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:19:56.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:56 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:19:56.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:56 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:19:56.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:56 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:19:56.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:56 smithi005 ceph-mon[29523]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:19:56.954 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:56 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:19:56.954 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:56 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:19:56.954 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:56 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:19:56.954 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:56 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:19:56.954 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:56 smithi186 ceph-mon[25720]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:19:59.280 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:59 smithi005 ceph-mon[24793]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:19:59.280 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:59 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:19:59.280 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:19:59 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:19:59.280 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:59 smithi005 ceph-mon[29523]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:19:59.280 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:59 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:19:59.280 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:19:59 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:19:59.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:59 smithi186 ceph-mon[25720]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:19:59.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:59 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:19:59.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:19:59 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:20:00.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:00 smithi005 ceph-mon[24793]: overall HEALTH_OK 2024-03-22T05:20:00.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:00 smithi005 ceph-mon[29523]: overall HEALTH_OK 2024-03-22T05:20:00.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:00 smithi186 ceph-mon[25720]: overall HEALTH_OK 2024-03-22T05:20:01.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:01 smithi005 ceph-mon[29523]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:20:01.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:01 smithi005 ceph-mon[24793]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:20:01.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:01 smithi186 ceph-mon[25720]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:20:02.678 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:20:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:20:02] "GET /metrics HTTP/1.1" 200 191458 "" "Prometheus/2.33.4" 2024-03-22T05:20:03.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:03 smithi005 ceph-mon[29523]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:20:03.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:03 smithi005 ceph-mon[24793]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:20:03.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:03 smithi186 ceph-mon[25720]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:20:03.294 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:20:03 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:20:03] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-22T05:20:04.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:04 smithi005 ceph-mon[24793]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:20:04.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:04 smithi005 ceph-mon[29523]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:20:04.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:04 smithi186 ceph-mon[25720]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:20:06.473 INFO:teuthology.orchestra.run.smithi005.stdout:107374182430 2024-03-22T05:20:06.474 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-22T05:20:07.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:07 smithi186 ceph-mon[25720]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:20:07.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:07 smithi005 ceph-mon[24793]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:20:07.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:07 smithi005 ceph-mon[29523]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:20:07.614 INFO:teuthology.orchestra.run.smithi005.stdout:231928233996 2024-03-22T05:20:07.614 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-22T05:20:09.273 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:09 smithi005 ceph-mon[29523]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:20:09.273 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:09 smithi005 ceph-mon[24793]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:20:09.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:09 smithi186 ceph-mon[25720]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:20:10.334 INFO:teuthology.orchestra.run.smithi005.stdout:128849018907 2024-03-22T05:20:10.335 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-22T05:20:10.935 INFO:teuthology.orchestra.run.smithi005.stdout:197568495633 2024-03-22T05:20:10.935 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-22T05:20:11.132 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:11 smithi005 ceph-mon[24793]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:20:11.132 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:11 smithi005 ceph-mon[29523]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:20:11.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:11 smithi186 ceph-mon[25720]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:20:11.581 INFO:teuthology.orchestra.run.smithi005.stdout:55834574889 2024-03-22T05:20:11.582 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-22T05:20:11.951 INFO:teuthology.orchestra.run.smithi005.stdout:34359738413 2024-03-22T05:20:11.951 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-22T05:20:12.523 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:20:12 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:20:12] "GET /metrics HTTP/1.1" 200 191458 "" "Prometheus/2.33.4" 2024-03-22T05:20:12.545 INFO:teuthology.orchestra.run.smithi005.stdout:77309411364 2024-03-22T05:20:12.545 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-22T05:20:13.122 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T05:20:13.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:13 smithi005 ceph-mon[24793]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:20:13.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:13 smithi005 ceph-mon[29523]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:20:13.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:13 smithi186 ceph-mon[25720]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:20:13.294 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:20:13 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:20:13] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-22T05:20:13.543 INFO:teuthology.orchestra.run.smithi005.stdout:163208757271 2024-03-22T05:20:13.543 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-22T05:20:13.800 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T05:20:13.804 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T05:20:13.805 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T05:20:14.061 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T05:20:15.006 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:14 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:20:15.006 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:14 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:20:15.006 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:14 smithi005 ceph-mon[24793]: Deploying daemon alertmanager.a on smithi005 2024-03-22T05:20:15.006 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:14 smithi005 ceph-mon[24793]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:20:15.007 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:14 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:20:15.007 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:14 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:20:15.007 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:14 smithi005 ceph-mon[29523]: Deploying daemon alertmanager.a on smithi005 2024-03-22T05:20:15.007 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:14 smithi005 ceph-mon[29523]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:20:15.241 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T05:20:15.242 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T05:20:15.244 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T05:20:15.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:14 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:20:15.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:14 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:20:15.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:14 smithi186 ceph-mon[25720]: Deploying daemon alertmanager.a on smithi005 2024-03-22T05:20:15.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:14 smithi186 ceph-mon[25720]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:20:17.186 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:17 smithi005 ceph-mon[24793]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:20:17.186 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:17 smithi005 ceph-mon[29523]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:20:17.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:17 smithi186 ceph-mon[25720]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:20:17.697 INFO:teuthology.orchestra.run.smithi005.stdout:231928233998 2024-03-22T05:20:18.136 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:18 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/3964678350' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-22T05:20:18.136 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:18 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/3964678350' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-22T05:20:18.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:18 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/3964678350' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-22T05:20:19.242 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:19 smithi005 ceph-mon[24793]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:20:19.242 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:19 smithi005 ceph-mon[29523]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:20:19.265 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928233996 got 231928233998 for osd.7 2024-03-22T05:20:19.265 DEBUG:teuthology.parallel:result is None 2024-03-22T05:20:19.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:19 smithi186 ceph-mon[25720]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:20:21.296 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:21 smithi186 ceph-mon[25720]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:20:21.385 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:21 smithi005 ceph-mon[24793]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:20:21.385 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:21 smithi005 ceph-mon[29523]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:20:21.537 INFO:teuthology.orchestra.run.smithi005.stdout:107374182433 2024-03-22T05:20:22.255 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:22 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/348957759' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-22T05:20:22.255 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:22 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/348957759' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-22T05:20:22.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:22 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/348957759' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-22T05:20:22.546 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:20:22 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:20:22] "GET /metrics HTTP/1.1" 200 191444 "" "Prometheus/2.33.4" 2024-03-22T05:20:23.152 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:23 smithi005 ceph-mon[24793]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:20:23.152 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:23 smithi005 ceph-mon[29523]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:20:23.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:23 smithi186 ceph-mon[25720]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:20:23.294 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:20:23 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:20:23] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-22T05:20:23.444 INFO:teuthology.orchestra.run.smithi005.stdout:128849018909 2024-03-22T05:20:24.258 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:24 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/3833904250' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-22T05:20:24.258 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:24 smithi005 ceph-mon[24793]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:20:24.258 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:24 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/3833904250' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-22T05:20:24.259 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:24 smithi005 ceph-mon[29523]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:20:24.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:24 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/3833904250' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-22T05:20:24.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:24 smithi186 ceph-mon[25720]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:20:26.775 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182430 got 107374182433 for osd.3 2024-03-22T05:20:26.775 DEBUG:teuthology.parallel:result is None 2024-03-22T05:20:26.889 INFO:teuthology.orchestra.run.smithi005.stdout:55834574893 2024-03-22T05:20:27.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:27 smithi005 ceph-mon[29523]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:20:27.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:27 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/1108974250' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-22T05:20:27.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:27 smithi005 ceph-mon[24793]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:20:27.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:27 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/1108974250' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-22T05:20:27.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:27 smithi186 ceph-mon[25720]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:20:27.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:27 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/1108974250' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-22T05:20:27.856 INFO:teuthology.orchestra.run.smithi005.stdout:163208757274 2024-03-22T05:20:28.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:28 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/586839888' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-22T05:20:28.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:28 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/586839888' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-22T05:20:28.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:28 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/586839888' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-22T05:20:29.119 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:29 smithi005 ceph-mon[29523]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:20:29.119 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:29 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-22T05:20:29.119 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:29 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-22T05:20:29.119 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:29 smithi005 ceph-mon[24793]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:20:29.119 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:29 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-22T05:20:29.119 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:29 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-22T05:20:29.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:29 smithi186 ceph-mon[25720]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:20:29.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:29 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-22T05:20:29.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:29 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-22T05:20:29.511 INFO:teuthology.orchestra.run.smithi005.stdout:197568495637 2024-03-22T05:20:30.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:30 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/2652430396' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-22T05:20:30.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:30 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/2652430396' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-22T05:20:30.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:30 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/2652430396' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-22T05:20:30.525 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495633 got 197568495637 for osd.6 2024-03-22T05:20:30.525 DEBUG:teuthology.parallel:result is None 2024-03-22T05:20:31.115 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:31 smithi005 ceph-mon[24793]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:20:31.116 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:31 smithi005 ceph-mon[29523]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:20:31.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:31 smithi186 ceph-mon[25720]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:20:31.382 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:20:31 smithi005 systemd[1]: Starting Ceph alertmanager.a for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:20:31.409 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849018907 got 128849018909 for osd.4 2024-03-22T05:20:31.410 DEBUG:teuthology.parallel:result is None 2024-03-22T05:20:32.279 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757271 got 163208757274 for osd.5 2024-03-22T05:20:32.279 DEBUG:teuthology.parallel:result is None 2024-03-22T05:20:32.511 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:20:32 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:20:32] "GET /metrics HTTP/1.1" 200 191448 "" "Prometheus/2.33.4" 2024-03-22T05:20:32.847 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574889 got 55834574893 for osd.1 2024-03-22T05:20:32.847 DEBUG:teuthology.parallel:result is None 2024-03-22T05:20:32.961 INFO:teuthology.orchestra.run.smithi005.stdout:77309411368 2024-03-22T05:20:33.010 INFO:teuthology.orchestra.run.smithi005.stdout:34359738418 2024-03-22T05:20:33.202 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:20:32 smithi005 podman[47391]: 2024-03-22 05:20:32.85619349 +0000 UTC m=+0.350964023 image pull ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4 quay.io/prometheus/alertmanager:v0.23.0 2024-03-22T05:20:33.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:33 smithi005 ceph-mon[24793]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:20:33.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:33 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/2018205939' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-22T05:20:33.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:33 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/2039288315' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-22T05:20:33.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:33 smithi005 ceph-mon[29523]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:20:33.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:33 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/2018205939' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-22T05:20:33.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:33 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/2039288315' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-22T05:20:33.294 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:20:33 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:20:33] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-22T05:20:33.295 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:33 smithi186 ceph-mon[25720]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:20:33.295 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:33 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/2018205939' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-22T05:20:33.295 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:33 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/2039288315' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-22T05:20:33.781 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738413 got 34359738418 for osd.0 2024-03-22T05:20:33.782 DEBUG:teuthology.parallel:result is None 2024-03-22T05:20:34.127 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:20:33 smithi005 podman[47391]: 2024-03-22 05:20:33.889849381 +0000 UTC m=+1.384619885 volume create b83eda3a53be39d99dcf8702c2961ce644f30d4fcba844e7cdb557215db65220 2024-03-22T05:20:34.127 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:20:33 smithi005 podman[47391]: 2024-03-22 05:20:33.956894652 +0000 UTC m=+1.451665202 container create eeff9b837b19a67465ef6aa958359bacf7f9ae0e50f8b79a37952c552d87df21 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-22T05:20:34.390 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:34 smithi005 ceph-mon[24793]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:20:34.390 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:34 smithi005 ceph-mon[29523]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:20:34.391 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:20:34 smithi005 podman[47391]: 2024-03-22 05:20:34.251471228 +0000 UTC m=+1.746241742 container init eeff9b837b19a67465ef6aa958359bacf7f9ae0e50f8b79a37952c552d87df21 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-22T05:20:34.391 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:20:34 smithi005 podman[47391]: 2024-03-22 05:20:34.254427784 +0000 UTC m=+1.749198295 container start eeff9b837b19a67465ef6aa958359bacf7f9ae0e50f8b79a37952c552d87df21 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-22T05:20:34.391 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:20:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[47482]: level=info ts=2024-03-22T05:20:34.266Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-03-22T05:20:34.391 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:20:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[47482]: level=info ts=2024-03-22T05:20:34.266Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-03-22T05:20:34.391 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:20:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[47482]: level=info ts=2024-03-22T05:20:34.267Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.5 port=9094 2024-03-22T05:20:34.391 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:20:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[47482]: level=info ts=2024-03-22T05:20:34.269Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-22T05:20:34.391 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:20:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[47482]: level=info ts=2024-03-22T05:20:34.295Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-22T05:20:34.391 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:20:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[47482]: level=info ts=2024-03-22T05:20:34.295Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-22T05:20:34.391 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:20:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[47482]: level=info ts=2024-03-22T05:20:34.298Z caller=main.go:518 msg=Listening address=:9093 2024-03-22T05:20:34.391 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:20:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[47482]: level=info ts=2024-03-22T05:20:34.298Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-03-22T05:20:34.391 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:20:34 smithi005 bash[47391]: eeff9b837b19a67465ef6aa958359bacf7f9ae0e50f8b79a37952c552d87df21 2024-03-22T05:20:34.392 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:20:34 smithi005 systemd[1]: Started Ceph alertmanager.a for f59f36e2-e80a-11ee-95cd-87774f69a715. 2024-03-22T05:20:34.498 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:34 smithi186 ceph-mon[25720]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:20:34.825 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411364 got 77309411368 for osd.2 2024-03-22T05:20:34.826 DEBUG:teuthology.parallel:result is None 2024-03-22T05:20:34.826 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-22T05:20:34.826 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-22T05:20:35.047 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T05:20:35.430 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:35 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:20:35.431 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:35 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:20:35.431 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:35 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:20:35.431 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:35 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-22T05:20:35.431 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:35 smithi005 ceph-mon[29523]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-22T05:20:35.431 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:35 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:20:35.431 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:35 smithi005 ceph-mon[29523]: Deploying daemon grafana.a on smithi186 2024-03-22T05:20:35.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:35 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:20:35.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:35 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:20:35.431 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:35 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:20:35.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:35 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-22T05:20:35.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:35 smithi005 ceph-mon[24793]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-22T05:20:35.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:35 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:20:35.432 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:35 smithi005 ceph-mon[24793]: Deploying daemon grafana.a on smithi186 2024-03-22T05:20:35.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:35 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:20:35.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:35 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:20:35.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:35 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:20:35.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:35 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-22T05:20:35.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:35 smithi186 ceph-mon[25720]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-22T05:20:35.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:35 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:20:35.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:35 smithi186 ceph-mon[25720]: Deploying daemon grafana.a on smithi186 2024-03-22T05:20:36.521 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:36 smithi005 ceph-mon[24793]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:20:36.521 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:20:36 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[47482]: level=info ts=2024-03-22T05:20:36.269Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000171806s 2024-03-22T05:20:36.521 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:36 smithi005 ceph-mon[29523]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:20:36.718 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-22T05:20:36.722 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-03-22T05:20:36.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:36 smithi186 ceph-mon[25720]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:20:37.451 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2024-03-22T05:20:35.972739+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48752,"kb_used_data":6064,"kb_used_omap":0,"kb_used_meta":42624,"kb_avail":749912464,"statfs":{"total":767960285184,"available":767910363136,"internally_reserved":0,"allocated":6209536,"data_stored":3486322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43646976},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002439"},"pg_stats":[{"pgid":"1.0","version":"59'128","reported_seq":262,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-22T05:19:28.066939+0000","last_change":"2024-03-22T05:19:15.153848+0000","last_active":"2024-03-22T05:19:28.066939+0000","last_peered":"2024-03-22T05:19:28.066939+0000","last_clean":"2024-03-22T05:19:28.066939+0000","last_became_active":"2024-03-22T05:19:15.153426+0000","last_became_peered":"2024-03-22T05:19:15.153426+0000","last_unstale":"2024-03-22T05:19:28.066939+0000","last_undegraded":"2024-03-22T05:19:28.066939+0000","last_fullsized":"2024-03-22T05:19:28.066939+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T05:17:17.510724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T05:17:17.510724+0000","last_clean_scrub_stamp":"2024-03-22T05:17:17.510724+0000","objects_scrubbed":0,"log_size":128,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:05:22.363175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234002,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6080,"kb_used_data":1080,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739072,"statfs":{"total":95995035648,"available":95988809728,"internally_reserved":0,"allocated":1105920,"data_stored":766128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 22 05:20:15 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.627,"5min":0.627,"15min":0.627},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.58399999999999996}]},{"osd":1,"last update":"Fri Mar 22 05:20:15 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.56100000000000005}]},{"osd":2,"last update":"Fri Mar 22 05:20:15 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.49099999999999999}]},{"osd":3,"last update":"Fri Mar 22 05:20:15 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.64600000000000002}]},{"osd":4,"last update":"Fri Mar 22 05:20:15 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.63300000000000001}]},{"osd":5,"last update":"Fri Mar 22 05:20:15 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.751}]},{"osd":6,"last update":"Fri Mar 22 05:20:15 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.71199999999999997}]}]},{"osd":6,"up_from":46,"seq":197568495639,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6148,"kb_used_data":1084,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739004,"statfs":{"total":95995035648,"available":95988740096,"internally_reserved":0,"allocated":1110016,"data_stored":766451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 22 05:19:48 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.754},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.70499999999999996}]},{"osd":1,"last update":"Fri Mar 22 05:19:48 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.66700000000000004}]},{"osd":2,"last update":"Fri Mar 22 05:19:48 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.67500000000000004}]},{"osd":3,"last update":"Fri Mar 22 05:19:48 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.65000000000000002}]},{"osd":4,"last update":"Fri Mar 22 05:19:48 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.442,"5min":0.442,"15min":0.442},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.625,"5min":0.625,"15min":0.625},"last":0.58599999999999997}]},{"osd":5,"last update":"Fri Mar 22 05:19:48 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.51000000000000001}]},{"osd":7,"last update":"Fri Mar 22 05:20:12 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.503},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.55200000000000005}]}]},{"osd":1,"up_from":13,"seq":55834574895,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6336,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738816,"statfs":{"total":95995035648,"available":95988547584,"internally_reserved":0,"allocated":581632,"data_stored":237788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 22 05:19:57 2024","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.34399999999999997,"15min":0.318},"min":{"1min":0.249,"5min":0.186,"15min":0.186},"max":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.34200000000000003,"15min":0.32000000000000001},"min":{"1min":0.23599999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":1.1739999999999999,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.31900000000000001}]},{"osd":2,"last update":"Fri Mar 22 05:20:22 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.41099999999999998,"15min":0.39100000000000001},"min":{"1min":0.27000000000000002,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":1.806,"5min":1.806,"15min":1.806},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.495,"5min":0.42299999999999999,"15min":0.40500000000000003},"min":{"1min":0.26400000000000001,"5min":0.247,"15min":0.247},"max":{"1min":2.0489999999999999,"5min":2.0489999999999999,"15min":2.0489999999999999},"last":0.39200000000000002}]},{"osd":3,"last update":"Fri Mar 22 05:19:43 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.41199999999999998,"15min":0.40799999999999997},"min":{"1min":0.32700000000000001,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.38400000000000001,"15min":0.378},"min":{"1min":0.26700000000000002,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.29299999999999998}]},{"osd":4,"last update":"Fri Mar 22 05:20:02 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.57099999999999995,"15min":0.56499999999999995},"min":{"1min":0.41599999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.2689999999999999,"5min":1.2689999999999999,"15min":1.2689999999999999},"last":0.89600000000000002},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.56000000000000005,"15min":0.55000000000000004},"min":{"1min":0.46600000000000003,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":1.3240000000000001,"5min":1.3240000000000001,"15min":1.3240000000000001},"last":0.66200000000000003}]},{"osd":5,"last update":"Fri Mar 22 05:20:24 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.57599999999999996,"15min":0.56200000000000006},"min":{"1min":0.46600000000000003,"5min":0.432,"15min":0.432},"max":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.59199999999999997,"15min":0.58199999999999996},"min":{"1min":0.46100000000000002,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.21,"5min":1.21,"15min":1.21},"last":0.64000000000000001}]},{"osd":6,"last update":"Fri Mar 22 05:19:46 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.80300000000000005}]},{"osd":7,"last update":"Fri Mar 22 05:20:13 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.86099999999999999},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.3500000000000001,"5min":1.3500000000000001,"15min":1.3500000000000001},"last":0.71499999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738419,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6912,"kb_used_data":1080,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738240,"statfs":{"total":95995035648,"available":95987957760,"internally_reserved":0,"allocated":1105920,"data_stored":766186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Mar 22 05:20:04 2024","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.34499999999999997,"15min":0.32900000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.53900000000000003,"5min":0.58299999999999996,"15min":0.58299999999999996},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.372,"5min":0.36499999999999999,"15min":0.34499999999999997},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.54100000000000004,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.35599999999999998}]},{"osd":2,"last update":"Fri Mar 22 05:20:29 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.40000000000000002,"15min":0.39600000000000002},"min":{"1min":0.27600000000000002,"5min":0.245,"15min":0.245},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.316},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.38600000000000001,"15min":0.379},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.375}]},{"osd":3,"last update":"Fri Mar 22 05:19:44 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.66400000000000003,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.44},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.46400000000000002,"15min":0.46600000000000003},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.56799999999999995,"5min":0.624,"15min":0.624},"last":0.28000000000000003}]},{"osd":4,"last update":"Fri Mar 22 05:20:09 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.57199999999999995,"15min":0.56999999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.502}]},{"osd":5,"last update":"Fri Mar 22 05:20:29 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.60599999999999998,"15min":0.60499999999999998},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.61699999999999999,"15min":0.61299999999999999},"min":{"1min":0.44500000000000001,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.51700000000000002}]},{"osd":6,"last update":"Fri Mar 22 05:19:50 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.56399999999999995}]},{"osd":7,"last update":"Fri Mar 22 05:20:15 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.59599999999999997}]}]},{"osd":2,"up_from":18,"seq":77309411369,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6204,"kb_used_data":564,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738948,"statfs":{"total":95995035648,"available":95988682752,"internally_reserved":0,"allocated":577536,"data_stored":237523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 22 05:20:20 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.374,"15min":0.371},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.61399999999999999,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.307},{"interface":"front","average":{"1min":0.373,"5min":0.377,"15min":0.36299999999999999},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.56799999999999995,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.32100000000000001}]},{"osd":1,"last update":"Fri Mar 22 05:20:20 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.377,"15min":0.36899999999999999},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.54700000000000004,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.379,"5min":0.40600000000000003,"15min":0.39700000000000002},"min":{"1min":0.27800000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.54400000000000004,"5min":0.621,"15min":0.621},"last":0.34100000000000003}]},{"osd":3,"last update":"Fri Mar 22 05:19:48 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.45000000000000001,"15min":0.44900000000000001},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.624,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.443,"5min":0.46700000000000003,"15min":0.47099999999999997},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.59599999999999997,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.35399999999999998}]},{"osd":4,"last update":"Fri Mar 22 05:20:05 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.59499999999999997,"15min":0.58999999999999997},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.625,"5min":0.60199999999999998,"15min":0.59799999999999998},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.77800000000000002,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.63300000000000001}]},{"osd":5,"last update":"Fri Mar 22 05:20:25 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.41899999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63100000000000001,"15min":0.63},"min":{"1min":0.44400000000000001,"5min":0.438,"15min":0.438},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.622}]},{"osd":6,"last update":"Fri Mar 22 05:19:48 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.45500000000000002}]},{"osd":7,"last update":"Fri Mar 22 05:20:12 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.502},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.64700000000000002}]}]},{"osd":3,"up_from":25,"seq":107374182436,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5692,"kb_used_data":564,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739460,"statfs":{"total":95995035648,"available":95989207040,"internally_reserved":0,"allocated":577536,"data_stored":237523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 22 05:19:47 2024","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.40799999999999997,"15min":0.41299999999999998},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.432,"15min":0.434},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.56899999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.35199999999999998}]},{"osd":1,"last update":"Fri Mar 22 05:19:47 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.45100000000000001,"15min":0.45700000000000002},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.56399999999999995,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.40300000000000002,"15min":0.39500000000000002},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.33000000000000002}]},{"osd":2,"last update":"Fri Mar 22 05:19:47 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.45700000000000002,"15min":0.45800000000000002},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.46100000000000002,"15min":0.46300000000000002},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.36499999999999999}]},{"osd":4,"last update":"Fri Mar 22 05:20:07 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.58999999999999997,"15min":0.58899999999999997},"min":{"1min":0.46000000000000002,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.59499999999999997,"15min":0.59199999999999997},"min":{"1min":0.47799999999999998,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.84499999999999997,"5min":1.038,"15min":1.038},"last":0.64100000000000001}]},{"osd":5,"last update":"Fri Mar 22 05:20:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.63200000000000001,"15min":0.64400000000000002},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.79800000000000004,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.627},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.61399999999999999,"15min":0.62},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.80900000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.60499999999999998}]},{"osd":6,"last update":"Fri Mar 22 05:19:47 2024","interfaces":[{"interface":"back","average":{"1min":1.4419999999999999,"5min":1.4419999999999999,"15min":1.4419999999999999},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":14.997999999999999,"5min":14.997999999999999,"15min":14.997999999999999},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.65700000000000003}]},{"osd":7,"last update":"Fri Mar 22 05:20:13 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.57199999999999995}]}]},{"osd":4,"up_from":30,"seq":128849018912,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5760,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739392,"statfs":{"total":95995035648,"available":95989137408,"internally_reserved":0,"allocated":581632,"data_stored":237788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 22 05:20:05 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.55400000000000005,"15min":0.55600000000000005},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.58199999999999996,"15min":0.58799999999999997},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.70399999999999996}]},{"osd":1,"last update":"Fri Mar 22 05:20:05 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.52900000000000003,"15min":0.53200000000000003},"min":{"1min":0.38600000000000001,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.72899999999999998,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.54200000000000004,"15min":0.54700000000000004},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.67000000000000004}]},{"osd":2,"last update":"Fri Mar 22 05:20:05 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.54500000000000004,"15min":0.54300000000000004},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.503,"5min":0.53400000000000003,"15min":0.53900000000000003},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.66400000000000003,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.69299999999999995}]},{"osd":3,"last update":"Fri Mar 22 05:20:05 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53700000000000003,"15min":0.53800000000000003},"min":{"1min":0.39700000000000002,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.69999999999999996,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.56499999999999995,"15min":0.56699999999999995},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.748,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.56299999999999994}]},{"osd":5,"last update":"Fri Mar 22 05:20:28 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.42999999999999999,"15min":0.41999999999999998},"min":{"1min":0.315,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.41299999999999998,"15min":0.40100000000000002},"min":{"1min":0.31,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.63900000000000001}]},{"osd":6,"last update":"Fri Mar 22 05:19:48 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.438,"15min":0.438},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.65600000000000003}]},{"osd":7,"last update":"Fri Mar 22 05:20:11 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.747}]}]},{"osd":5,"up_from":38,"seq":163208757276,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5620,"kb_used_data":556,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739532,"statfs":{"total":95995035648,"available":95989280768,"internally_reserved":0,"allocated":569344,"data_stored":236935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 22 05:20:27 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.52900000000000003,"15min":0.53100000000000003},"min":{"1min":0.32600000000000001,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.71799999999999997,"5min":0.751,"15min":0.751},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.52500000000000002,"15min":0.52700000000000002},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.74099999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.624}]},{"osd":1,"last update":"Fri Mar 22 05:20:27 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.52400000000000002,"15min":0.52800000000000002},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.79000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.52600000000000002,"15min":0.52500000000000002},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.51700000000000002}]},{"osd":2,"last update":"Fri Mar 22 05:20:27 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.54100000000000004,"15min":0.54400000000000004},"min":{"1min":0.377,"5min":0.37,"15min":0.37},"max":{"1min":0.69499999999999995,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.54200000000000004,"15min":0.54400000000000004},"min":{"1min":0.35999999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.54600000000000004}]},{"osd":3,"last update":"Fri Mar 22 05:20:27 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.72399999999999998,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.56299999999999994,"15min":0.56699999999999995},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.69499999999999995,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.49299999999999999}]},{"osd":4,"last update":"Fri Mar 22 05:20:27 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.40699999999999997,"15min":0.40100000000000002},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.42199999999999999,"15min":0.41599999999999998},"min":{"1min":0.31,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.441}]},{"osd":6,"last update":"Fri Mar 22 05:19:48 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.57299999999999995}]},{"osd":7,"last update":"Fri Mar 22 05:20:13 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.55900000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-22T05:20:37.453 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-22T05:20:37.675 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T05:20:37.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:37 smithi005 ceph-mon[24793]: from='client.24514 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-22T05:20:37.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:37 smithi005 ceph-mon[29523]: from='client.24514 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-22T05:20:37.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:37 smithi186 ceph-mon[25720]: from='client.24514 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-22T05:20:39.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:39 smithi186 ceph-mon[25720]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:20:39.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:39 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:20:39.397 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-22T05:20:39.397 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:39 smithi005 ceph-mon[24793]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:20:39.398 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:39 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:20:39.398 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:39 smithi005 ceph-mon[29523]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:20:39.398 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:39 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:20:39.399 INFO:teuthology.orchestra.run.smithi005.stderr:dumped all 2024-03-22T05:20:39.999 INFO:teuthology.orchestra.run.smithi005.stdout:{"pg_ready":true,"pg_map":{"version":38,"stamp":"2024-03-22T05:20:37.973147+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48752,"kb_used_data":6064,"kb_used_omap":0,"kb_used_meta":42624,"kb_avail":749912464,"statfs":{"total":767960285184,"available":767910363136,"internally_reserved":0,"allocated":6209536,"data_stored":3486322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43646976},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002515"},"pg_stats":[{"pgid":"1.0","version":"59'128","reported_seq":262,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-22T05:19:28.066939+0000","last_change":"2024-03-22T05:19:15.153848+0000","last_active":"2024-03-22T05:19:28.066939+0000","last_peered":"2024-03-22T05:19:28.066939+0000","last_clean":"2024-03-22T05:19:28.066939+0000","last_became_active":"2024-03-22T05:19:15.153426+0000","last_became_peered":"2024-03-22T05:19:15.153426+0000","last_unstale":"2024-03-22T05:19:28.066939+0000","last_undegraded":"2024-03-22T05:19:28.066939+0000","last_fullsized":"2024-03-22T05:19:28.066939+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-22T05:17:17.510724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-22T05:17:17.510724+0000","last_clean_scrub_stamp":"2024-03-22T05:17:17.510724+0000","objects_scrubbed":0,"log_size":128,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-23T11:05:22.363175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234002,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6080,"kb_used_data":1080,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739072,"statfs":{"total":95995035648,"available":95988809728,"internally_reserved":0,"allocated":1105920,"data_stored":766128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 22 05:20:15 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.627,"5min":0.627,"15min":0.627},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.58399999999999996}]},{"osd":1,"last update":"Fri Mar 22 05:20:15 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.56100000000000005}]},{"osd":2,"last update":"Fri Mar 22 05:20:15 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.49099999999999999}]},{"osd":3,"last update":"Fri Mar 22 05:20:15 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.64600000000000002}]},{"osd":4,"last update":"Fri Mar 22 05:20:15 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.63300000000000001}]},{"osd":5,"last update":"Fri Mar 22 05:20:15 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.751}]},{"osd":6,"last update":"Fri Mar 22 05:20:15 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.71199999999999997}]}]},{"osd":6,"up_from":46,"seq":197568495639,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6148,"kb_used_data":1084,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739004,"statfs":{"total":95995035648,"available":95988740096,"internally_reserved":0,"allocated":1110016,"data_stored":766451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 22 05:19:48 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.754},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.70499999999999996}]},{"osd":1,"last update":"Fri Mar 22 05:19:48 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.66700000000000004}]},{"osd":2,"last update":"Fri Mar 22 05:19:48 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.67500000000000004}]},{"osd":3,"last update":"Fri Mar 22 05:19:48 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.65000000000000002}]},{"osd":4,"last update":"Fri Mar 22 05:19:48 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.442,"5min":0.442,"15min":0.442},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.625,"5min":0.625,"15min":0.625},"last":0.58599999999999997}]},{"osd":5,"last update":"Fri Mar 22 05:19:48 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.51000000000000001}]},{"osd":7,"last update":"Fri Mar 22 05:20:12 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.503},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.55200000000000005}]}]},{"osd":1,"up_from":13,"seq":55834574895,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6336,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738816,"statfs":{"total":95995035648,"available":95988547584,"internally_reserved":0,"allocated":581632,"data_stored":237788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 22 05:19:57 2024","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.34399999999999997,"15min":0.318},"min":{"1min":0.249,"5min":0.186,"15min":0.186},"max":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.34200000000000003,"15min":0.32000000000000001},"min":{"1min":0.23599999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":1.1739999999999999,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.31900000000000001}]},{"osd":2,"last update":"Fri Mar 22 05:20:22 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.41099999999999998,"15min":0.39100000000000001},"min":{"1min":0.27000000000000002,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":1.806,"5min":1.806,"15min":1.806},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.495,"5min":0.42299999999999999,"15min":0.40500000000000003},"min":{"1min":0.26400000000000001,"5min":0.247,"15min":0.247},"max":{"1min":2.0489999999999999,"5min":2.0489999999999999,"15min":2.0489999999999999},"last":0.39200000000000002}]},{"osd":3,"last update":"Fri Mar 22 05:19:43 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.41199999999999998,"15min":0.40799999999999997},"min":{"1min":0.32700000000000001,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.38400000000000001,"15min":0.378},"min":{"1min":0.26700000000000002,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.29299999999999998}]},{"osd":4,"last update":"Fri Mar 22 05:20:02 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.57099999999999995,"15min":0.56499999999999995},"min":{"1min":0.41599999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.2689999999999999,"5min":1.2689999999999999,"15min":1.2689999999999999},"last":0.89600000000000002},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.56000000000000005,"15min":0.55000000000000004},"min":{"1min":0.46600000000000003,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":1.3240000000000001,"5min":1.3240000000000001,"15min":1.3240000000000001},"last":0.66200000000000003}]},{"osd":5,"last update":"Fri Mar 22 05:20:24 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.57599999999999996,"15min":0.56200000000000006},"min":{"1min":0.46600000000000003,"5min":0.432,"15min":0.432},"max":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.59199999999999997,"15min":0.58199999999999996},"min":{"1min":0.46100000000000002,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.21,"5min":1.21,"15min":1.21},"last":0.64000000000000001}]},{"osd":6,"last update":"Fri Mar 22 05:19:46 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.80300000000000005}]},{"osd":7,"last update":"Fri Mar 22 05:20:13 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.86099999999999999},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.3500000000000001,"5min":1.3500000000000001,"15min":1.3500000000000001},"last":0.71499999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738419,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6912,"kb_used_data":1080,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738240,"statfs":{"total":95995035648,"available":95987957760,"internally_reserved":0,"allocated":1105920,"data_stored":766186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Mar 22 05:20:04 2024","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.34499999999999997,"15min":0.32900000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.53900000000000003,"5min":0.58299999999999996,"15min":0.58299999999999996},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.372,"5min":0.36499999999999999,"15min":0.34499999999999997},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.54100000000000004,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.35599999999999998}]},{"osd":2,"last update":"Fri Mar 22 05:20:29 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.40000000000000002,"15min":0.39600000000000002},"min":{"1min":0.27600000000000002,"5min":0.245,"15min":0.245},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.316},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.38600000000000001,"15min":0.379},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.375}]},{"osd":3,"last update":"Fri Mar 22 05:19:44 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.66400000000000003,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.44},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.46400000000000002,"15min":0.46600000000000003},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.56799999999999995,"5min":0.624,"15min":0.624},"last":0.28000000000000003}]},{"osd":4,"last update":"Fri Mar 22 05:20:09 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.57199999999999995,"15min":0.56999999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.502}]},{"osd":5,"last update":"Fri Mar 22 05:20:29 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.60599999999999998,"15min":0.60499999999999998},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.61699999999999999,"15min":0.61299999999999999},"min":{"1min":0.44500000000000001,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.51700000000000002}]},{"osd":6,"last update":"Fri Mar 22 05:19:50 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.56399999999999995}]},{"osd":7,"last update":"Fri Mar 22 05:20:15 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.59599999999999997}]}]},{"osd":2,"up_from":18,"seq":77309411369,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6204,"kb_used_data":564,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738948,"statfs":{"total":95995035648,"available":95988682752,"internally_reserved":0,"allocated":577536,"data_stored":237523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 22 05:20:20 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.374,"15min":0.371},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.61399999999999999,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.307},{"interface":"front","average":{"1min":0.373,"5min":0.377,"15min":0.36299999999999999},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.56799999999999995,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.32100000000000001}]},{"osd":1,"last update":"Fri Mar 22 05:20:20 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.377,"15min":0.36899999999999999},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.54700000000000004,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.379,"5min":0.40600000000000003,"15min":0.39700000000000002},"min":{"1min":0.27800000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.54400000000000004,"5min":0.621,"15min":0.621},"last":0.34100000000000003}]},{"osd":3,"last update":"Fri Mar 22 05:19:48 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.45000000000000001,"15min":0.44900000000000001},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.624,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.443,"5min":0.46700000000000003,"15min":0.47099999999999997},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.59599999999999997,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.35399999999999998}]},{"osd":4,"last update":"Fri Mar 22 05:20:05 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.59499999999999997,"15min":0.58999999999999997},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.625,"5min":0.60199999999999998,"15min":0.59799999999999998},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.77800000000000002,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.63300000000000001}]},{"osd":5,"last update":"Fri Mar 22 05:20:25 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.41899999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63100000000000001,"15min":0.63},"min":{"1min":0.44400000000000001,"5min":0.438,"15min":0.438},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.622}]},{"osd":6,"last update":"Fri Mar 22 05:19:48 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.45500000000000002}]},{"osd":7,"last update":"Fri Mar 22 05:20:12 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.502},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.64700000000000002}]}]},{"osd":3,"up_from":25,"seq":107374182436,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5692,"kb_used_data":564,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739460,"statfs":{"total":95995035648,"available":95989207040,"internally_reserved":0,"allocated":577536,"data_stored":237523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 22 05:19:47 2024","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.40799999999999997,"15min":0.41299999999999998},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.432,"15min":0.434},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.56899999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.35199999999999998}]},{"osd":1,"last update":"Fri Mar 22 05:19:47 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.45100000000000001,"15min":0.45700000000000002},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.56399999999999995,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.40300000000000002,"15min":0.39500000000000002},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.33000000000000002}]},{"osd":2,"last update":"Fri Mar 22 05:19:47 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.45700000000000002,"15min":0.45800000000000002},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.46100000000000002,"15min":0.46300000000000002},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.36499999999999999}]},{"osd":4,"last update":"Fri Mar 22 05:20:07 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.58999999999999997,"15min":0.58899999999999997},"min":{"1min":0.46000000000000002,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.59499999999999997,"15min":0.59199999999999997},"min":{"1min":0.47799999999999998,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.84499999999999997,"5min":1.038,"15min":1.038},"last":0.64100000000000001}]},{"osd":5,"last update":"Fri Mar 22 05:20:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.63200000000000001,"15min":0.64400000000000002},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.79800000000000004,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.627},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.61399999999999999,"15min":0.62},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.80900000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.60499999999999998}]},{"osd":6,"last update":"Fri Mar 22 05:19:47 2024","interfaces":[{"interface":"back","average":{"1min":1.4419999999999999,"5min":1.4419999999999999,"15min":1.4419999999999999},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":14.997999999999999,"5min":14.997999999999999,"15min":14.997999999999999},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.65700000000000003}]},{"osd":7,"last update":"Fri Mar 22 05:20:13 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.57199999999999995}]}]},{"osd":4,"up_from":30,"seq":128849018912,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5760,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739392,"statfs":{"total":95995035648,"available":95989137408,"internally_reserved":0,"allocated":581632,"data_stored":237788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 22 05:20:05 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.55400000000000005,"15min":0.55600000000000005},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.58199999999999996,"15min":0.58799999999999997},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.70399999999999996}]},{"osd":1,"last update":"Fri Mar 22 05:20:05 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.52900000000000003,"15min":0.53200000000000003},"min":{"1min":0.38600000000000001,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.72899999999999998,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.54200000000000004,"15min":0.54700000000000004},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.67000000000000004}]},{"osd":2,"last update":"Fri Mar 22 05:20:05 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.54500000000000004,"15min":0.54300000000000004},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.503,"5min":0.53400000000000003,"15min":0.53900000000000003},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.66400000000000003,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.69299999999999995}]},{"osd":3,"last update":"Fri Mar 22 05:20:05 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53700000000000003,"15min":0.53800000000000003},"min":{"1min":0.39700000000000002,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.69999999999999996,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.56499999999999995,"15min":0.56699999999999995},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.748,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.56299999999999994}]},{"osd":5,"last update":"Fri Mar 22 05:20:28 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.42999999999999999,"15min":0.41999999999999998},"min":{"1min":0.315,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.41299999999999998,"15min":0.40100000000000002},"min":{"1min":0.31,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.63900000000000001}]},{"osd":6,"last update":"Fri Mar 22 05:19:48 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.438,"15min":0.438},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.65600000000000003}]},{"osd":7,"last update":"Fri Mar 22 05:20:11 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.747}]}]},{"osd":5,"up_from":38,"seq":163208757276,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5620,"kb_used_data":556,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739532,"statfs":{"total":95995035648,"available":95989280768,"internally_reserved":0,"allocated":569344,"data_stored":236935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Mar 22 05:20:27 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.52900000000000003,"15min":0.53100000000000003},"min":{"1min":0.32600000000000001,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.71799999999999997,"5min":0.751,"15min":0.751},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.52500000000000002,"15min":0.52700000000000002},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.74099999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.624}]},{"osd":1,"last update":"Fri Mar 22 05:20:27 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.52400000000000002,"15min":0.52800000000000002},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.79000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.52600000000000002,"15min":0.52500000000000002},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.51700000000000002}]},{"osd":2,"last update":"Fri Mar 22 05:20:27 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.54100000000000004,"15min":0.54400000000000004},"min":{"1min":0.377,"5min":0.37,"15min":0.37},"max":{"1min":0.69499999999999995,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.54200000000000004,"15min":0.54400000000000004},"min":{"1min":0.35999999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.54600000000000004}]},{"osd":3,"last update":"Fri Mar 22 05:20:27 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.72399999999999998,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.56299999999999994,"15min":0.56699999999999995},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.69499999999999995,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.49299999999999999}]},{"osd":4,"last update":"Fri Mar 22 05:20:27 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.40699999999999997,"15min":0.40100000000000002},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.42199999999999999,"15min":0.41599999999999998},"min":{"1min":0.31,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.441}]},{"osd":6,"last update":"Fri Mar 22 05:19:48 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.57299999999999995}]},{"osd":7,"last update":"Fri Mar 22 05:20:13 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.55900000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-22T05:20:40.000 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-22T05:20:40.001 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-22T05:20:40.001 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-22T05:20:40.001 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -- ceph health --format=json 2024-03-22T05:20:40.224 INFO:teuthology.orchestra.run.smithi005.stderr:Inferring config /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/mon.a/config 2024-03-22T05:20:41.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:41 smithi186 ceph-mon[25720]: from='client.14610 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-22T05:20:41.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:41 smithi186 ceph-mon[25720]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:20:41.323 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:41 smithi005 ceph-mon[29523]: from='client.14610 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-22T05:20:41.323 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:41 smithi005 ceph-mon[29523]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:20:41.323 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:41 smithi005 ceph-mon[24793]: from='client.14610 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-22T05:20:41.323 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:41 smithi005 ceph-mon[24793]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:20:41.877 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-22T05:20:41.877 INFO:teuthology.orchestra.run.smithi005.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-22T05:20:42.195 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:42 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/734247596' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-22T05:20:42.195 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:42 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/734247596' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-22T05:20:42.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:42 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/734247596' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-22T05:20:42.487 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-22T05:20:42.487 INFO:tasks.cephadm:Setup complete, yielding 2024-03-22T05:20:42.487 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-22T05:20:42.498 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi005.front.sepia.ceph.com 2024-03-22T05:20:42.498 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'radosgw-admin realm create --rgw-realm=r --default' 2024-03-22T05:20:42.702 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:20:42 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:20:42] "GET /metrics HTTP/1.1" 200 191448 "" "Prometheus/2.33.4" 2024-03-22T05:20:43.116 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:43 smithi005 ceph-mon[24793]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:20:43.116 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:43 smithi005 ceph-mon[29523]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:20:43.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:43 smithi186 ceph-mon[25720]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:20:43.294 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:20:43 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:20:43] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-22T05:20:44.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:44 smithi005 ceph-mon[29523]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:20:44.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:44 smithi005 ceph-mon[24793]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:20:44.703 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:20:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[47482]: level=info ts=2024-03-22T05:20:44.271Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.00189829s 2024-03-22T05:20:44.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:44 smithi186 ceph-mon[25720]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:20:45.331 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-03-22T05:20:45.331 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "9bf819c1-7665-4d37-b4e2-f78c50dd5f3e", 2024-03-22T05:20:45.331 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "r", 2024-03-22T05:20:45.331 INFO:teuthology.orchestra.run.smithi005.stdout: "current_period": "1207cb3d-1f26-4c53-9152-ccd96f5df313", 2024-03-22T05:20:45.331 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1 2024-03-22T05:20:45.331 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-03-22T05:20:45.628 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:45 smithi005 ceph-mon[29523]: osdmap e60: 8 total, 8 up, 8 in 2024-03-22T05:20:45.628 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:45 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/2405130844' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-22T05:20:45.628 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:45 smithi005 ceph-mon[29523]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-22T05:20:45.628 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:45 smithi005 ceph-mon[29523]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-22T05:20:45.628 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:45 smithi005 ceph-mon[29523]: osdmap e61: 8 total, 8 up, 8 in 2024-03-22T05:20:45.628 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:45 smithi005 ceph-mon[24793]: osdmap e60: 8 total, 8 up, 8 in 2024-03-22T05:20:45.628 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:45 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/2405130844' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-22T05:20:45.628 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:45 smithi005 ceph-mon[24793]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-22T05:20:45.628 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:45 smithi005 ceph-mon[24793]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-22T05:20:45.629 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:45 smithi005 ceph-mon[24793]: osdmap e61: 8 total, 8 up, 8 in 2024-03-22T05:20:45.715 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:45 smithi186 ceph-mon[25720]: osdmap e60: 8 total, 8 up, 8 in 2024-03-22T05:20:45.716 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:45 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/2405130844' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-22T05:20:45.716 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:45 smithi186 ceph-mon[25720]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-22T05:20:45.716 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:45 smithi186 ceph-mon[25720]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-22T05:20:45.716 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:45 smithi186 ceph-mon[25720]: osdmap e61: 8 total, 8 up, 8 in 2024-03-22T05:20:45.940 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'radosgw-admin zonegroup create --rgw-zonegroup=default --master --default' 2024-03-22T05:20:46.572 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:46 smithi005 ceph-mon[29523]: pgmap v44: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:20:46.573 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:46 smithi005 ceph-mon[24793]: pgmap v44: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:20:46.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:46 smithi186 ceph-mon[25720]: pgmap v44: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:20:47.304 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:47 smithi005 ceph-mon[24793]: osdmap e62: 8 total, 8 up, 8 in 2024-03-22T05:20:47.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:47 smithi186 ceph-mon[25720]: osdmap e62: 8 total, 8 up, 8 in 2024-03-22T05:20:47.612 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:47 smithi005 ceph-mon[29523]: osdmap e62: 8 total, 8 up, 8 in 2024-03-22T05:20:47.728 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-03-22T05:20:47.729 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "4faa26b9-70eb-48ae-81f5-634fd700721a", 2024-03-22T05:20:47.729 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "default", 2024-03-22T05:20:47.729 INFO:teuthology.orchestra.run.smithi005.stdout: "api_name": "default", 2024-03-22T05:20:47.729 INFO:teuthology.orchestra.run.smithi005.stdout: "is_master": "true", 2024-03-22T05:20:47.729 INFO:teuthology.orchestra.run.smithi005.stdout: "endpoints": [], 2024-03-22T05:20:47.729 INFO:teuthology.orchestra.run.smithi005.stdout: "hostnames": [], 2024-03-22T05:20:47.729 INFO:teuthology.orchestra.run.smithi005.stdout: "hostnames_s3website": [], 2024-03-22T05:20:47.729 INFO:teuthology.orchestra.run.smithi005.stdout: "master_zone": "", 2024-03-22T05:20:47.729 INFO:teuthology.orchestra.run.smithi005.stdout: "zones": [], 2024-03-22T05:20:47.729 INFO:teuthology.orchestra.run.smithi005.stdout: "placement_targets": [], 2024-03-22T05:20:47.729 INFO:teuthology.orchestra.run.smithi005.stdout: "default_placement": "", 2024-03-22T05:20:47.729 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_id": "9bf819c1-7665-4d37-b4e2-f78c50dd5f3e", 2024-03-22T05:20:47.729 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_policy": { 2024-03-22T05:20:47.729 INFO:teuthology.orchestra.run.smithi005.stdout: "groups": [] 2024-03-22T05:20:47.730 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-03-22T05:20:47.730 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-03-22T05:20:48.399 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:48 smithi005 ceph-mon[29523]: pgmap v46: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:20:48.399 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:48 smithi005 ceph-mon[24793]: pgmap v46: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:20:48.427 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default' 2024-03-22T05:20:48.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-mon[25720]: pgmap v46: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:20:48.544 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 systemd[1]: Starting Ceph grafana.a for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:20:48.826 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 podman[41775]: 2024-03-22 05:20:48.471239124 +0000 UTC m=+0.017103851 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-03-22T05:20:48.826 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 podman[41775]: 2024-03-22 05:20:48.59272814 +0000 UTC m=+0.138592904 container create 27324e98a8bba35073c6b755d0d7719ad44aecb9c43c027b09f3ba3cd51f4edf (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a, io.openshift.expose-services=, architecture=x86_64, description=Ceph Grafana Container, com.redhat.component=ubi8-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.openshift.tags=base rhel8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Paul Cuzner , build-date=2022-03-28T10:36:18.413762, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.24.2, vendor=Red Hat, Inc., name=ubi8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-type=git, release=236.1648460182, summary=Grafana Container configured for Ceph mgr/dashboard integration, version=8.5, io.k8s.display-name=Red Hat Universal Base Image 8) 2024-03-22T05:20:48.826 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 podman[41775]: 2024-03-22 05:20:48.827007224 +0000 UTC m=+0.372871953 container init 27324e98a8bba35073c6b755d0d7719ad44aecb9c43c027b09f3ba3cd51f4edf (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a, io.openshift.expose-services=, com.redhat.component=ubi8-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base rhel8, summary=Grafana Container configured for Ceph mgr/dashboard integration, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8.5, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, name=ubi8, vcs-type=git, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, architecture=x86_64, build-date=2022-03-28T10:36:18.413762, description=Ceph Grafana Container, maintainer=Paul Cuzner , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, distribution-scope=public, io.buildah.version=1.24.2, io.k8s.display-name=Red Hat Universal Base Image 8, release=236.1648460182) 2024-03-22T05:20:49.077 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 podman[41775]: 2024-03-22 05:20:48.830272102 +0000 UTC m=+0.376136835 container start 27324e98a8bba35073c6b755d0d7719ad44aecb9c43c027b09f3ba3cd51f4edf (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a, io.openshift.expose-services=, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=ubi8, maintainer=Paul Cuzner , io.buildah.version=1.24.2, io.k8s.display-name=Red Hat Universal Base Image 8, release=236.1648460182, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=ubi8-container, build-date=2022-03-28T10:36:18.413762, io.openshift.tags=base rhel8, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, summary=Grafana Container configured for Ceph mgr/dashboard integration, version=8.5, architecture=x86_64, description=Ceph Grafana Container, vendor=Red Hat, Inc., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI) 2024-03-22T05:20:49.077 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="The state of unified alerting is still not defined. The decision will be made during as we run the database migrations" logger=settings 2024-03-22T05:20:49.077 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=warn msg="falling back to legacy setting of 'min_interval_seconds'; please use the configuration option in the `unified_alerting` section if Grafana 8 alerts are enabled." logger=settings 2024-03-22T05:20:49.077 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Config loaded from" logger=settings file=/usr/share/grafana/conf/defaults.ini 2024-03-22T05:20:49.077 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Config loaded from" logger=settings file=/etc/grafana/grafana.ini 2024-03-22T05:20:49.077 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_DATA=/var/lib/grafana" 2024-03-22T05:20:49.077 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_LOGS=/var/log/grafana" 2024-03-22T05:20:49.077 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-03-22T05:20:49.077 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-03-22T05:20:49.077 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Path Home" logger=settings path=/usr/share/grafana 2024-03-22T05:20:49.078 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Path Data" logger=settings path=/var/lib/grafana 2024-03-22T05:20:49.078 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Path Logs" logger=settings path=/var/log/grafana 2024-03-22T05:20:49.078 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Path Plugins" logger=settings path=/var/lib/grafana/plugins 2024-03-22T05:20:49.079 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Path Provisioning" logger=settings path=/etc/grafana/provisioning 2024-03-22T05:20:49.079 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="App mode production" logger=settings 2024-03-22T05:20:49.079 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3 2024-03-22T05:20:49.079 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=warn msg="SQLite database file has broader permissions than it should" logger=sqlstore path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-03-22T05:20:49.079 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Starting DB migrations" logger=migrator 2024-03-22T05:20:49.079 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="create migration_log table" 2024-03-22T05:20:49.079 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="create user table" 2024-03-22T05:20:49.079 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.login" 2024-03-22T05:20:49.079 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.email" 2024-03-22T05:20:49.080 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_login - v1" 2024-03-22T05:20:49.080 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_email - v1" 2024-03-22T05:20:49.080 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table user to user_v1 - v1" 2024-03-22T05:20:49.080 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="create user table v2" 2024-03-22T05:20:49.080 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_login - v2" 2024-03-22T05:20:49.080 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_email - v2" 2024-03-22T05:20:49.080 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2024-03-22T05:20:49.080 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table user_v1" 2024-03-22T05:20:49.080 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="Add column help_flags1 to user table" 2024-03-22T05:20:49.080 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="Update user table charset" 2024-03-22T05:20:49.080 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="Add last_seen_at column to user" 2024-03-22T05:20:49.080 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="Add missing user data" 2024-03-22T05:20:49.080 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="Add is_disabled column to user" 2024-03-22T05:20:49.081 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="Add index user.login/user.email" 2024-03-22T05:20:49.081 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="Add is_service_account column to user" 2024-03-22T05:20:49.081 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="create temp user table v1-7" 2024-03-22T05:20:49.081 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_email - v1-7" 2024-03-22T05:20:49.081 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_org_id - v1-7" 2024-03-22T05:20:49.081 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_code - v1-7" 2024-03-22T05:20:49.081 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_status - v1-7" 2024-03-22T05:20:49.081 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="Update temp_user table charset" 2024-03-22T05:20:49.081 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_email - v1" 2024-03-22T05:20:49.081 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_org_id - v1" 2024-03-22T05:20:49.081 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_code - v1" 2024-03-22T05:20:49.081 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_status - v1" 2024-03-22T05:20:49.081 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-03-22T05:20:49.081 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="create temp_user v2" 2024-03-22T05:20:49.081 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_email - v2" 2024-03-22T05:20:49.082 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_org_id - v2" 2024-03-22T05:20:49.082 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_code - v2" 2024-03-22T05:20:49.082 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_status - v2" 2024-03-22T05:20:49.082 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="copy temp_user v1 to v2" 2024-03-22T05:20:49.082 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="drop temp_user_tmp_qwerty" 2024-03-22T05:20:49.082 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="Set created for temp users that will otherwise prematurely expire" 2024-03-22T05:20:49.082 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="create star table" 2024-03-22T05:20:49.082 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index star.user_id_dashboard_id" 2024-03-22T05:20:49.082 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="create org table v1" 2024-03-22T05:20:49.082 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_name - v1" 2024-03-22T05:20:49.082 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="create org_user table v1" 2024-03-22T05:20:49.082 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 bash[41775]: 27324e98a8bba35073c6b755d0d7719ad44aecb9c43c027b09f3ba3cd51f4edf 2024-03-22T05:20:49.082 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_org_user_org_id - v1" 2024-03-22T05:20:49.082 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_user_org_id_user_id - v1" 2024-03-22T05:20:49.083 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_org_user_user_id - v1" 2024-03-22T05:20:49.083 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 systemd[1]: Started Ceph grafana.a for f59f36e2-e80a-11ee-95cd-87774f69a715. 2024-03-22T05:20:49.083 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="Update org table charset" 2024-03-22T05:20:49.083 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="Update org_user table charset" 2024-03-22T05:20:49.083 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate all Read Only Viewers to Viewers" 2024-03-22T05:20:49.083 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard table" 2024-03-22T05:20:49.083 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard.account_id" 2024-03-22T05:20:49.083 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_account_id_slug" 2024-03-22T05:20:49.083 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_tag table" 2024-03-22T05:20:49.083 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_tag.dasboard_id_term" 2024-03-22T05:20:49.083 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-03-22T05:20:49.084 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard to dashboard_v1 - v1" 2024-03-22T05:20:49.084 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard v2" 2024-03-22T05:20:49.084 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_org_id - v2" 2024-03-22T05:20:49.084 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_org_id_slug - v2" 2024-03-22T05:20:49.084 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard v1 to v2" 2024-03-22T05:20:49.085 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_v1" 2024-03-22T05:20:49.085 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard.data to mediumtext v1" 2024-03-22T05:20:49.085 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="Add column updated_by in dashboard - v2" 2024-03-22T05:20:49.085 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="Add column created_by in dashboard - v2" 2024-03-22T05:20:49.085 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="Add column gnetId in dashboard" 2024-03-22T05:20:49.085 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for gnetId in dashboard" 2024-03-22T05:20:49.085 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_id in dashboard" 2024-03-22T05:20:49.085 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for plugin_id in dashboard" 2024-03-22T05:20:49.085 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_id in dashboard_tag" 2024-03-22T05:20:49.086 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard table charset" 2024-03-22T05:20:49.086 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_tag table charset" 2024-03-22T05:20:49.086 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="Add column folder_id in dashboard" 2024-03-22T05:20:49.086 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="Add column isFolder in dashboard" 2024-03-22T05:20:49.086 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="Add column has_acl in dashboard" 2024-03-22T05:20:49.086 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in dashboard" 2024-03-22T05:20:49.086 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in dashboard" 2024-03-22T05:20:49.086 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index dashboard_org_id_uid" 2024-03-22T05:20:49.086 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_slug" 2024-03-22T05:20:49.086 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard title length" 2024-03-22T05:20:49.086 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index for dashboard_org_id_title_folder_id" 2024-03-22T05:20:49.086 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning" 2024-03-22T05:20:49.087 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-03-22T05:20:49.087 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning v2" 2024-03-22T05:20:49.087 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-03-22T05:20:49.087 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-03-22T05:20:49.087 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard_provisioning v1 to v2" 2024-03-22T05:20:49.087 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="drop dashboard_provisioning_tmp_qwerty" 2024-03-22T05:20:49.087 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="Add check_sum column" 2024-03-22T05:20:49.087 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_title" 2024-03-22T05:20:49.087 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="delete tags for deleted dashboards" 2024-03-22T05:20:49.087 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:48+0000 lvl=info msg="Executing migration" logger=migrator id="delete stars for deleted dashboards" 2024-03-22T05:20:49.087 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_is_folder" 2024-03-22T05:20:49.087 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table" 2024-03-22T05:20:49.087 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add index data_source.account_id" 2024-03-22T05:20:49.088 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index data_source.account_id_name" 2024-03-22T05:20:49.088 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_data_source_account_id - v1" 2024-03-22T05:20:49.088 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_data_source_account_id_name - v1" 2024-03-22T05:20:49.088 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table data_source to data_source_v1 - v1" 2024-03-22T05:20:49.089 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table v2" 2024-03-22T05:20:49.089 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_data_source_org_id - v2" 2024-03-22T05:20:49.089 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_data_source_org_id_name - v2" 2024-03-22T05:20:49.089 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2024-03-22T05:20:49.089 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table data_source_v1 #2" 2024-03-22T05:20:49.089 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Add column with_credentials" 2024-03-22T05:20:49.089 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Add secure json data column" 2024-03-22T05:20:49.090 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Update data_source table charset" 2024-03-22T05:20:49.090 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Update initial version to 1" 2024-03-22T05:20:49.090 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Add read_only data column" 2024-03-22T05:20:49.090 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate logging ds to loki ds" 2024-03-22T05:20:49.090 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Update json_data with nulls" 2024-03-22T05:20:49.090 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Add uid column" 2024-03-22T05:20:49.090 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid value" 2024-03-22T05:20:49.090 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index datasource_org_id_uid" 2024-03-22T05:20:49.090 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index datasource_org_id_is_default" 2024-03-22T05:20:49.090 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table" 2024-03-22T05:20:49.090 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id" 2024-03-22T05:20:49.090 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.key" 2024-03-22T05:20:49.091 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id_name" 2024-03-22T05:20:49.091 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_api_key_account_id - v1" 2024-03-22T05:20:49.091 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_key - v1" 2024-03-22T05:20:49.091 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_account_id_name - v1" 2024-03-22T05:20:49.091 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table api_key to api_key_v1 - v1" 2024-03-22T05:20:49.091 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table v2" 2024-03-22T05:20:49.091 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_api_key_org_id - v2" 2024-03-22T05:20:49.091 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_key - v2" 2024-03-22T05:20:49.091 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_org_id_name - v2" 2024-03-22T05:20:49.091 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="copy api_key v1 to v2" 2024-03-22T05:20:49.091 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table api_key_v1" 2024-03-22T05:20:49.091 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Update api_key table charset" 2024-03-22T05:20:49.091 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Add expires to api_key table" 2024-03-22T05:20:49.091 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Add service account foreign key" 2024-03-22T05:20:49.092 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v4" 2024-03-22T05:20:49.092 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_snapshot_v4 #1" 2024-03-22T05:20:49.092 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v5 #2" 2024-03-22T05:20:49.092 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_key - v5" 2024-03-22T05:20:49.092 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-03-22T05:20:49.092 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_snapshot_user_id - v5" 2024-03-22T05:20:49.092 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_snapshot to mediumtext v2" 2024-03-22T05:20:49.092 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_snapshot table charset" 2024-03-22T05:20:49.092 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external_delete_url to dashboard_snapshots table" 2024-03-22T05:20:49.092 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Add encrypted dashboard json column" 2024-03-22T05:20:49.092 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-03-22T05:20:49.092 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="create quota table v1" 2024-03-22T05:20:49.092 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_quota_org_id_user_id_target - v1" 2024-03-22T05:20:49.092 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Update quota table charset" 2024-03-22T05:20:49.093 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="create plugin_setting table" 2024-03-22T05:20:49.093 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-03-22T05:20:49.093 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_version to plugin_settings" 2024-03-22T05:20:49.328 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Update plugin_setting table charset" 2024-03-22T05:20:49.328 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="create session table" 2024-03-22T05:20:49.329 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist table" 2024-03-22T05:20:49.329 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist_item table" 2024-03-22T05:20:49.329 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist table v2" 2024-03-22T05:20:49.329 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist item table v2" 2024-03-22T05:20:49.329 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist table charset" 2024-03-22T05:20:49.329 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist_item table charset" 2024-03-22T05:20:49.329 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v2" 2024-03-22T05:20:49.329 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v3" 2024-03-22T05:20:49.329 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="create preferences table v3" 2024-03-22T05:20:49.330 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Update preferences table charset" 2024-03-22T05:20:49.330 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Add column team_id in preferences" 2024-03-22T05:20:49.330 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Update team_id column values in preferences" 2024-03-22T05:20:49.330 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Add column week_start in preferences" 2024-03-22T05:20:49.330 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="create alert table v1" 2024-03-22T05:20:49.330 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert org_id & id " 2024-03-22T05:20:49.330 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert state" 2024-03-22T05:20:49.330 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert dashboard_id" 2024-03-22T05:20:49.330 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v1" 2024-03-22T05:20:49.331 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-03-22T05:20:49.331 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-03-22T05:20:49.331 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-03-22T05:20:49.331 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v2" 2024-03-22T05:20:49.331 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-03-22T05:20:49.332 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="copy alert_rule_tag v1 to v2" 2024-03-22T05:20:49.332 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="drop table alert_rule_tag_v1" 2024-03-22T05:20:49.332 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification table v1" 2024-03-22T05:20:49.332 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Add column is_default" 2024-03-22T05:20:49.332 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Add column frequency" 2024-03-22T05:20:49.332 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Add column send_reminder" 2024-03-22T05:20:49.332 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Add column disable_resolve_message" 2024-03-22T05:20:49.332 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification org_id & name" 2024-03-22T05:20:49.332 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert table charset" 2024-03-22T05:20:49.332 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert_notification table charset" 2024-03-22T05:20:49.332 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="create notification_journal table v1" 2024-03-22T05:20:49.332 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add index notification_journal org_id & alert_id & notifier_id" 2024-03-22T05:20:49.332 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_notification_journal" 2024-03-22T05:20:49.332 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification_state table v1" 2024-03-22T05:20:49.333 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-03-22T05:20:49.333 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Add for to alert table" 2024-03-22T05:20:49.333 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in alert_notification" 2024-03-22T05:20:49.333 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in alert_notification" 2024-03-22T05:20:49.333 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_notification_org_id_uid" 2024-03-22T05:20:49.333 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_name" 2024-03-22T05:20:49.333 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Add column secure_settings in alert_notification" 2024-03-22T05:20:49.333 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert.settings to mediumtext" 2024-03-22T05:20:49.333 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Add non-unique index alert_notification_state_alert_id" 2024-03-22T05:20:49.333 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Add non-unique index alert_rule_tag_alert_id" 2024-03-22T05:20:49.333 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old annotation table v4" 2024-03-22T05:20:49.333 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="create annotation table v5" 2024-03-22T05:20:49.334 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 0 v3" 2024-03-22T05:20:49.334 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 1 v3" 2024-03-22T05:20:49.334 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 2 v3" 2024-03-22T05:20:49.334 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 3 v3" 2024-03-22T05:20:49.334 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 4 v3" 2024-03-22T05:20:49.334 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Update annotation table charset" 2024-03-22T05:20:49.334 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Add column region_id to annotation table" 2024-03-22T05:20:49.334 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Drop category_id index" 2024-03-22T05:20:49.334 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Add column tags to annotation table" 2024-03-22T05:20:49.334 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v2" 2024-03-22T05:20:49.335 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index annotation_tag.annotation_id_tag_id" 2024-03-22T05:20:49.335 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-03-22T05:20:49.335 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-03-22T05:20:49.335 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v3" 2024-03-22T05:20:49.335 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-03-22T05:20:49.335 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="copy annotation_tag v2 to v3" 2024-03-22T05:20:49.335 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="drop table annotation_tag_v2" 2024-03-22T05:20:49.335 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert annotations and set TEXT to empty" 2024-03-22T05:20:49.335 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Add created time to annotation table" 2024-03-22T05:20:49.335 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Add updated time to annotation table" 2024-03-22T05:20:49.335 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for created in annotation table" 2024-03-22T05:20:49.335 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for updated in annotation table" 2024-03-22T05:20:49.335 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Convert existing annotations from seconds to milliseconds" 2024-03-22T05:20:49.335 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Add epoch_end column" 2024-03-22T05:20:49.336 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for epoch_end" 2024-03-22T05:20:49.336 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Make epoch_end the same as epoch" 2024-03-22T05:20:49.336 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Move region to single row" 2024-03-22T05:20:49.336 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch from annotation table" 2024-03-22T05:20:49.336 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-03-22T05:20:49.336 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-03-22T05:20:49.336 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_epoch_end_epoch on annotation table" 2024-03-22T05:20:49.336 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch_epoch_end from annotation table" 2024-03-22T05:20:49.336 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for alert_id on annotation table" 2024-03-22T05:20:49.336 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="create test_data table" 2024-03-22T05:20:49.336 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_version table v1" 2024-03-22T05:20:49.336 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_version.dashboard_id" 2024-03-22T05:20:49.337 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-03-22T05:20:49.337 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Set dashboard version to 1 where 0" 2024-03-22T05:20:49.337 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="save existing dashboard data in dashboard_version table v1" 2024-03-22T05:20:49.337 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_version.data to mediumtext v1" 2024-03-22T05:20:49.337 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="create team table" 2024-03-22T05:20:49.337 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add index team.org_id" 2024-03-22T05:20:49.337 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_org_id_name" 2024-03-22T05:20:49.337 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="create team member table" 2024-03-22T05:20:49.337 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.org_id" 2024-03-22T05:20:49.338 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_member_org_id_team_id_user_id" 2024-03-22T05:20:49.338 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.team_id" 2024-03-22T05:20:49.338 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Add column email to team table" 2024-03-22T05:20:49.338 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external to team_member table" 2024-03-22T05:20:49.338 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Add column permission to team_member table" 2024-03-22T05:20:49.338 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard acl table" 2024-03-22T05:20:49.338 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_dashboard_id" 2024-03-22T05:20:49.338 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_user_id" 2024-03-22T05:20:49.338 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_team_id" 2024-03-22T05:20:49.338 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_user_id" 2024-03-22T05:20:49.338 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_team_id" 2024-03-22T05:20:49.338 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_org_id_role" 2024-03-22T05:20:49.339 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_permission" 2024-03-22T05:20:49.339 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="save default acl rules in dashboard_acl table" 2024-03-22T05:20:49.339 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="delete acl rules for deleted dashboards and folders" 2024-03-22T05:20:49.339 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="create tag table" 2024-03-22T05:20:49.339 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add index tag.key_value" 2024-03-22T05:20:49.339 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="create login attempt table" 2024-03-22T05:20:49.339 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add index login_attempt.username" 2024-03-22T05:20:49.339 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_login_attempt_username - v1" 2024-03-22T05:20:49.339 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-03-22T05:20:49.339 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="create login_attempt v2" 2024-03-22T05:20:49.340 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_login_attempt_username - v2" 2024-03-22T05:20:49.341 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="copy login_attempt v1 to v2" 2024-03-22T05:20:49.341 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="drop login_attempt_tmp_qwerty" 2024-03-22T05:20:49.341 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth table" 2024-03-22T05:20:49.341 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-03-22T05:20:49.341 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="alter user_auth.auth_id to length 190" 2024-03-22T05:20:49.341 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth access token to user_auth" 2024-03-22T05:20:49.341 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth refresh token to user_auth" 2024-03-22T05:20:49.341 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth token type to user_auth" 2024-03-22T05:20:49.341 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth expiry to user_auth" 2024-03-22T05:20:49.341 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Add index to user_id column in user_auth" 2024-03-22T05:20:49.341 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="create server_lock table" 2024-03-22T05:20:49.341 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add index server_lock.operation_uid" 2024-03-22T05:20:49.341 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth token table" 2024-03-22T05:20:49.342 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.auth_token" 2024-03-22T05:20:49.342 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.prev_auth_token" 2024-03-22T05:20:49.342 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_auth_token.user_id" 2024-03-22T05:20:49.342 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Add revoked_at to the user auth token" 2024-03-22T05:20:49.342 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="create cache_data table" 2024-03-22T05:20:49.342 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index cache_data.cache_key" 2024-03-22T05:20:49.342 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="create short_url table v1" 2024-03-22T05:20:49.342 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add index short_url.org_id-uid" 2024-03-22T05:20:49.342 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="delete alert_definition table" 2024-03-22T05:20:49.342 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="recreate alert_definition table" 2024-03-22T05:20:49.342 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition on org_id and title columns" 2024-03-22T05:20:49.342 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition on org_id and uid columns" 2024-03-22T05:20:49.342 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_definition table data column to mediumtext in mysql" 2024-03-22T05:20:49.342 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="drop index in alert_definition on org_id and title columns" 2024-03-22T05:20:49.343 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="drop index in alert_definition on org_id and uid columns" 2024-03-22T05:20:49.343 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index in alert_definition on org_id and title columns" 2024-03-22T05:20:49.343 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index in alert_definition on org_id and uid columns" 2024-03-22T05:20:49.343 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Add column paused in alert_definition" 2024-03-22T05:20:49.343 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_definition table" 2024-03-22T05:20:49.343 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="delete alert_definition_version table" 2024-03-22T05:20:49.343 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="recreate alert_definition_version table" 2024-03-22T05:20:49.343 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-03-22T05:20:49.343 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-03-22T05:20:49.343 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_definition_version table data column to mediumtext in mysql" 2024-03-22T05:20:49.343 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_definition_version table" 2024-03-22T05:20:49.344 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_instance table" 2024-03-22T05:20:49.344 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-03-22T05:20:49.344 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_instance table on def_org_id, current_state columns" 2024-03-22T05:20:49.344 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add column current_state_end to alert_instance" 2024-03-22T05:20:49.344 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-03-22T05:20:49.344 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="remove index def_org_id, current_state on alert_instance" 2024-03-22T05:20:49.344 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="rename def_org_id to rule_org_id in alert_instance" 2024-03-22T05:20:49.344 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="rename def_uid to rule_uid in alert_instance" 2024-03-22T05:20:49.344 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-03-22T05:20:49.344 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add index rule_org_id, current_state on alert_instance" 2024-03-22T05:20:49.344 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_rule table" 2024-03-22T05:20:49.344 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id and title columns" 2024-03-22T05:20:49.344 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id and uid columns" 2024-03-22T05:20:49.344 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-03-22T05:20:49.345 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_rule table data column to mediumtext in mysql" 2024-03-22T05:20:49.345 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add column for to alert_rule" 2024-03-22T05:20:49.345 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add column annotations to alert_rule" 2024-03-22T05:20:49.345 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add column labels to alert_rule" 2024-03-22T05:20:49.345 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="remove unique index from alert_rule on org_id, title columns" 2024-03-22T05:20:49.345 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-03-22T05:20:49.345 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add dashboard_uid column to alert_rule" 2024-03-22T05:20:49.345 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add panel_id column to alert_rule" 2024-03-22T05:20:49.345 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-03-22T05:20:49.345 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_rule_version table" 2024-03-22T05:20:49.345 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-03-22T05:20:49.345 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-03-22T05:20:49.346 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_rule_version table data column to mediumtext in mysql" 2024-03-22T05:20:49.346 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add column for to alert_rule_version" 2024-03-22T05:20:49.346 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add column annotations to alert_rule_version" 2024-03-22T05:20:49.346 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add column labels to alert_rule_version" 2024-03-22T05:20:49.346 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id=create_alert_configuration_table 2024-03-22T05:20:49.346 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Add column default in alert_configuration" 2024-03-22T05:20:49.346 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-03-22T05:20:49.346 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add column org_id in alert_configuration" 2024-03-22T05:20:49.347 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_configuration table on org_id column" 2024-03-22T05:20:49.347 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id=create_ngalert_configuration_table 2024-03-22T05:20:49.347 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add index in ngalert_configuration on org_id column" 2024-03-22T05:20:49.347 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="clear migration entry \"remove unified alerting data\"" 2024-03-22T05:20:49.347 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="move dashboard alerts to unified alerting" 2024-03-22T05:20:49.347 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="create library_element table v1" 2024-03-22T05:20:49.347 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add index library_element org_id-folder_id-name-kind" 2024-03-22T05:20:49.348 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="create library_element_connection table v1" 2024-03-22T05:20:49.348 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add index library_element_connection element_id-kind-connection_id" 2024-03-22T05:20:49.348 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index library_element org_id_uid" 2024-03-22T05:20:49.348 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="clone move dashboard alerts to unified alerting" 2024-03-22T05:20:49.348 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="create data_keys table" 2024-03-22T05:20:49.348 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="create kv_store table v1" 2024-03-22T05:20:49.348 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add index kv_store.org_id-namespace-key" 2024-03-22T05:20:49.348 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="update dashboard_uid and panel_id from existing annotations" 2024-03-22T05:20:49.348 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="create permission table" 2024-03-22T05:20:49.348 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index permission.role_id" 2024-03-22T05:20:49.348 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role_id_action_scope" 2024-03-22T05:20:49.349 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="create role table" 2024-03-22T05:20:49.349 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add column display_name" 2024-03-22T05:20:49.349 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add column group_name" 2024-03-22T05:20:49.349 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add index role.org_id" 2024-03-22T05:20:49.349 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role_org_id_name" 2024-03-22T05:20:49.349 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add index role_org_id_uid" 2024-03-22T05:20:49.349 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="create team role table" 2024-03-22T05:20:49.599 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_role.org_id" 2024-03-22T05:20:49.599 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_role_org_id_team_id_role_id" 2024-03-22T05:20:49.599 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_role.team_id" 2024-03-22T05:20:49.600 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="create user role table" 2024-03-22T05:20:49.600 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_role.org_id" 2024-03-22T05:20:49.600 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_role_org_id_user_id_role_id" 2024-03-22T05:20:49.600 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_role.user_id" 2024-03-22T05:20:49.600 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="create builtin role table" 2024-03-22T05:20:49.600 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.role_id" 2024-03-22T05:20:49.600 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.name" 2024-03-22T05:20:49.600 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Add column org_id to builtin_role table" 2024-03-22T05:20:49.600 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.org_id" 2024-03-22T05:20:49.600 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index builtin_role_org_id_role_id_role" 2024-03-22T05:20:49.600 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index role_org_id_uid" 2024-03-22T05:20:49.600 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role.uid" 2024-03-22T05:20:49.600 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="create seed assignment table" 2024-03-22T05:20:49.600 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index builtin_role_role_name" 2024-03-22T05:20:49.601 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="migrations completed" logger=migrator performed=381 skipped=0 duration=516.960953ms 2024-03-22T05:20:49.601 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Created default organization" logger=sqlstore 2024-03-22T05:20:49.601 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Initialising plugins" logger=plugin.manager 2024-03-22T05:20:49.601 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=input 2024-03-22T05:20:49.601 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=grafana-piechart-panel 2024-03-22T05:20:49.601 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=vonage-status-panel 2024-03-22T05:20:49.601 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="Live Push Gateway initialization" logger=live.push_http 2024-03-22T05:20:49.601 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=warn msg="[Deprecated] the datasource provisioning config is outdated. please upgrade" logger=provisioning.datasources filename=/etc/grafana/provisioning/datasources/ceph-dashboard.yml 2024-03-22T05:20:49.601 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Dashboard1 uid=P43CA22E17D0F9596 2024-03-22T05:20:49.601 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="warming cache for startup" logger=ngalert 2024-03-22T05:20:49.601 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=https subUrl= socket= 2024-03-22T05:20:49.601 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:20:49+0000 lvl=info msg="starting MultiOrg Alertmanager" logger=ngalert.multiorg.alertmanager 2024-03-22T05:20:50.158 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-03-22T05:20:50.159 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "97eb5063-a634-49fc-801b-0370c4f7e8c9", 2024-03-22T05:20:50.159 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "z", 2024-03-22T05:20:50.159 INFO:teuthology.orchestra.run.smithi005.stdout: "domain_root": "z.rgw.meta:root", 2024-03-22T05:20:50.159 INFO:teuthology.orchestra.run.smithi005.stdout: "control_pool": "z.rgw.control", 2024-03-22T05:20:50.159 INFO:teuthology.orchestra.run.smithi005.stdout: "gc_pool": "z.rgw.log:gc", 2024-03-22T05:20:50.159 INFO:teuthology.orchestra.run.smithi005.stdout: "lc_pool": "z.rgw.log:lc", 2024-03-22T05:20:50.159 INFO:teuthology.orchestra.run.smithi005.stdout: "log_pool": "z.rgw.log", 2024-03-22T05:20:50.159 INFO:teuthology.orchestra.run.smithi005.stdout: "intent_log_pool": "z.rgw.log:intent", 2024-03-22T05:20:50.159 INFO:teuthology.orchestra.run.smithi005.stdout: "usage_log_pool": "z.rgw.log:usage", 2024-03-22T05:20:50.159 INFO:teuthology.orchestra.run.smithi005.stdout: "roles_pool": "z.rgw.meta:roles", 2024-03-22T05:20:50.159 INFO:teuthology.orchestra.run.smithi005.stdout: "reshard_pool": "z.rgw.log:reshard", 2024-03-22T05:20:50.159 INFO:teuthology.orchestra.run.smithi005.stdout: "user_keys_pool": "z.rgw.meta:users.keys", 2024-03-22T05:20:50.159 INFO:teuthology.orchestra.run.smithi005.stdout: "user_email_pool": "z.rgw.meta:users.email", 2024-03-22T05:20:50.160 INFO:teuthology.orchestra.run.smithi005.stdout: "user_swift_pool": "z.rgw.meta:users.swift", 2024-03-22T05:20:50.160 INFO:teuthology.orchestra.run.smithi005.stdout: "user_uid_pool": "z.rgw.meta:users.uid", 2024-03-22T05:20:50.160 INFO:teuthology.orchestra.run.smithi005.stdout: "otp_pool": "z.rgw.otp", 2024-03-22T05:20:50.160 INFO:teuthology.orchestra.run.smithi005.stdout: "system_key": { 2024-03-22T05:20:50.160 INFO:teuthology.orchestra.run.smithi005.stdout: "access_key": "", 2024-03-22T05:20:50.160 INFO:teuthology.orchestra.run.smithi005.stdout: "secret_key": "" 2024-03-22T05:20:50.160 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:20:50.160 INFO:teuthology.orchestra.run.smithi005.stdout: "placement_pools": [ 2024-03-22T05:20:50.160 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-03-22T05:20:50.160 INFO:teuthology.orchestra.run.smithi005.stdout: "key": "default-placement", 2024-03-22T05:20:50.160 INFO:teuthology.orchestra.run.smithi005.stdout: "val": { 2024-03-22T05:20:50.160 INFO:teuthology.orchestra.run.smithi005.stdout: "index_pool": "z.rgw.buckets.index", 2024-03-22T05:20:50.160 INFO:teuthology.orchestra.run.smithi005.stdout: "storage_classes": { 2024-03-22T05:20:50.160 INFO:teuthology.orchestra.run.smithi005.stdout: "STANDARD": { 2024-03-22T05:20:50.161 INFO:teuthology.orchestra.run.smithi005.stdout: "data_pool": "z.rgw.buckets.data" 2024-03-22T05:20:50.161 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-03-22T05:20:50.161 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:20:50.161 INFO:teuthology.orchestra.run.smithi005.stdout: "data_extra_pool": "z.rgw.buckets.non-ec", 2024-03-22T05:20:50.161 INFO:teuthology.orchestra.run.smithi005.stdout: "index_type": 0 2024-03-22T05:20:50.161 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-03-22T05:20:50.161 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-03-22T05:20:50.161 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-03-22T05:20:50.161 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_id": "9bf819c1-7665-4d37-b4e2-f78c50dd5f3e", 2024-03-22T05:20:50.161 INFO:teuthology.orchestra.run.smithi005.stdout: "notif_pool": "z.rgw.log:notif" 2024-03-22T05:20:50.161 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-03-22T05:20:50.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:49 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:20:50.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:49 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:20:50.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:49 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:20:50.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:49 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:20:50.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:49 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:20:50.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:49 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:20:50.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:49 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:20:50.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:49 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:20:50.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:20:50.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:20:50.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:20:50.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:49 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:20:50.721 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'radosgw-admin period update --rgw-realm=r --commit' 2024-03-22T05:20:51.252 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:50 smithi186 ceph-mon[25720]: pgmap v47: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 1.8 KiB/s wr, 2 op/s 2024-03-22T05:20:51.268 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:50 smithi005 ceph-mon[24793]: pgmap v47: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 1.8 KiB/s wr, 2 op/s 2024-03-22T05:20:51.269 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:50 smithi005 ceph-mon[29523]: pgmap v47: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 1.8 KiB/s wr, 2 op/s 2024-03-22T05:20:52.642 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:20:52 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:20:52] "GET /metrics HTTP/1.1" 200 191451 "" "Prometheus/2.33.4" 2024-03-22T05:20:53.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:53 smithi005 ceph-mon[29523]: pgmap v48: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 532 B/s rd, 1.4 KiB/s wr, 2 op/s 2024-03-22T05:20:53.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:53 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:20:53.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:53 smithi005 ceph-mon[24793]: pgmap v48: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 532 B/s rd, 1.4 KiB/s wr, 2 op/s 2024-03-22T05:20:53.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:53 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:20:53.294 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:20:53 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:20:53] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-22T05:20:53.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:53 smithi186 ceph-mon[25720]: pgmap v48: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 532 B/s rd, 1.4 KiB/s wr, 2 op/s 2024-03-22T05:20:53.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:53 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:20:54.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:54 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:20:54.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:54 smithi186 ceph-mon[25720]: osdmap e63: 8 total, 8 up, 8 in 2024-03-22T05:20:54.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:54 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/2230087265' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-03-22T05:20:54.332 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:54 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:20:54.333 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:54 smithi005 ceph-mon[24793]: osdmap e63: 8 total, 8 up, 8 in 2024-03-22T05:20:54.333 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:54 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/2230087265' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-03-22T05:20:54.333 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:54 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:20:54.333 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:54 smithi005 ceph-mon[29523]: osdmap e63: 8 total, 8 up, 8 in 2024-03-22T05:20:54.333 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:54 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/2230087265' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-03-22T05:20:55.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:55 smithi186 ceph-mon[25720]: pgmap v50: 65 pgs: 32 creating+peering, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 2.0 KiB/s wr, 6 op/s 2024-03-22T05:20:55.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:55 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/2230087265' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-03-22T05:20:55.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:55 smithi186 ceph-mon[25720]: osdmap e64: 8 total, 8 up, 8 in 2024-03-22T05:20:55.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:55 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:20:55.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:55 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:20:55.392 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:55 smithi005 ceph-mon[29523]: pgmap v50: 65 pgs: 32 creating+peering, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 2.0 KiB/s wr, 6 op/s 2024-03-22T05:20:55.393 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:55 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/2230087265' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-03-22T05:20:55.393 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:55 smithi005 ceph-mon[29523]: osdmap e64: 8 total, 8 up, 8 in 2024-03-22T05:20:55.393 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:55 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:20:55.393 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:55 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:20:55.394 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:55 smithi005 ceph-mon[24793]: pgmap v50: 65 pgs: 32 creating+peering, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 2.0 KiB/s wr, 6 op/s 2024-03-22T05:20:55.394 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:55 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/2230087265' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-03-22T05:20:55.394 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:55 smithi005 ceph-mon[24793]: osdmap e64: 8 total, 8 up, 8 in 2024-03-22T05:20:55.394 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:55 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:20:55.394 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:55 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:20:56.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:56 smithi186 ceph-mon[25720]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-22T05:20:56.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:56 smithi186 ceph-mon[25720]: Reconfiguring daemon alertmanager.a on smithi005 2024-03-22T05:20:56.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:56 smithi186 ceph-mon[25720]: osdmap e65: 8 total, 8 up, 8 in 2024-03-22T05:20:56.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:56 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/2230087265' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-03-22T05:20:56.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:56 smithi005 ceph-mon[24793]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-22T05:20:56.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:56 smithi005 ceph-mon[24793]: Reconfiguring daemon alertmanager.a on smithi005 2024-03-22T05:20:56.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:56 smithi005 ceph-mon[24793]: osdmap e65: 8 total, 8 up, 8 in 2024-03-22T05:20:56.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:56 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/2230087265' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-03-22T05:20:56.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:56 smithi005 ceph-mon[29523]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-22T05:20:56.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:56 smithi005 ceph-mon[29523]: Reconfiguring daemon alertmanager.a on smithi005 2024-03-22T05:20:56.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:56 smithi005 ceph-mon[29523]: osdmap e65: 8 total, 8 up, 8 in 2024-03-22T05:20:56.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:56 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/2230087265' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-03-22T05:20:57.120 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:57 smithi005 ceph-mon[24793]: pgmap v53: 97 pgs: 32 unknown, 32 creating+peering, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 4.3 KiB/s rd, 853 B/s wr, 5 op/s 2024-03-22T05:20:57.121 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:57 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/2230087265' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-03-22T05:20:57.121 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:57 smithi005 ceph-mon[24793]: osdmap e66: 8 total, 8 up, 8 in 2024-03-22T05:20:57.121 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:57 smithi005 ceph-mon[29523]: pgmap v53: 97 pgs: 32 unknown, 32 creating+peering, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 4.3 KiB/s rd, 853 B/s wr, 5 op/s 2024-03-22T05:20:57.121 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:57 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/2230087265' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-03-22T05:20:57.121 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:57 smithi005 ceph-mon[29523]: osdmap e66: 8 total, 8 up, 8 in 2024-03-22T05:20:57.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:57 smithi186 ceph-mon[25720]: pgmap v53: 97 pgs: 32 unknown, 32 creating+peering, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 4.3 KiB/s rd, 853 B/s wr, 5 op/s 2024-03-22T05:20:57.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:57 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/2230087265' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-03-22T05:20:57.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:57 smithi186 ceph-mon[25720]: osdmap e66: 8 total, 8 up, 8 in 2024-03-22T05:20:58.253 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:20:58 smithi005 systemd[1]: Stopping Ceph alertmanager.a for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:20:58.254 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:20:58 smithi005 bash[49212]: Error: no container with name or ID "ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager.a" found: no such container 2024-03-22T05:20:58.546 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:20:58 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[47482]: level=info ts=2024-03-22T05:20:58.253Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-03-22T05:20:58.546 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:20:58 smithi005 podman[49225]: 2024-03-22 05:20:58.274632568 +0000 UTC m=+0.123208379 container died eeff9b837b19a67465ef6aa958359bacf7f9ae0e50f8b79a37952c552d87df21 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-22T05:20:58.546 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:58 smithi005 ceph-mon[24793]: osdmap e67: 8 total, 8 up, 8 in 2024-03-22T05:20:58.546 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:58 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/971354302' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-03-22T05:20:58.546 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:58 smithi005 ceph-mon[24793]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-03-22T05:20:58.546 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:58 smithi005 ceph-mon[24793]: pgmap v56: 129 pgs: 64 unknown, 32 creating+peering, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:20:58.547 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:58 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:20:58.547 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:58 smithi005 ceph-mon[29523]: osdmap e67: 8 total, 8 up, 8 in 2024-03-22T05:20:58.547 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:58 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/971354302' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-03-22T05:20:58.547 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:58 smithi005 ceph-mon[29523]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-03-22T05:20:58.547 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:58 smithi005 ceph-mon[29523]: pgmap v56: 129 pgs: 64 unknown, 32 creating+peering, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:20:58.547 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:58 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:20:58.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:58 smithi186 ceph-mon[25720]: osdmap e67: 8 total, 8 up, 8 in 2024-03-22T05:20:58.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:58 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/971354302' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-03-22T05:20:58.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:58 smithi186 ceph-mon[25720]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-03-22T05:20:58.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:58 smithi186 ceph-mon[25720]: pgmap v56: 129 pgs: 64 unknown, 32 creating+peering, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:20:58.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:58 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:20:58.913 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:20:58 smithi005 podman[49225]: 2024-03-22 05:20:58.704345922 +0000 UTC m=+0.552921731 container cleanup eeff9b837b19a67465ef6aa958359bacf7f9ae0e50f8b79a37952c552d87df21 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-22T05:20:58.913 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:20:58 smithi005 bash[49225]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a 2024-03-22T05:20:58.913 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:20:58 smithi005 bash[49252]: Error: no container with name or ID "ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager.a" found: no such container 2024-03-22T05:20:59.202 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:20:58 smithi005 podman[49239]: 2024-03-22 05:20:58.91398562 +0000 UTC m=+0.633266329 container remove eeff9b837b19a67465ef6aa958359bacf7f9ae0e50f8b79a37952c552d87df21 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-22T05:20:59.202 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:20:58 smithi005 podman[49239]: 2024-03-22 05:20:58.998016308 +0000 UTC m=+0.717296989 volume remove b83eda3a53be39d99dcf8702c2961ce644f30d4fcba844e7cdb557215db65220 2024-03-22T05:20:59.202 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:20:59 smithi005 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-22T05:20:59.202 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:20:59 smithi005 systemd[1]: Stopped Ceph alertmanager.a for f59f36e2-e80a-11ee-95cd-87774f69a715. 2024-03-22T05:20:59.202 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:20:59 smithi005 systemd[1]: Starting Ceph alertmanager.a for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:20:59.515 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:59 smithi005 ceph-mon[29523]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-03-22T05:20:59.515 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:59 smithi005 ceph-mon[29523]: osdmap e68: 8 total, 8 up, 8 in 2024-03-22T05:20:59.516 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:59 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/971354302' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-22T05:20:59.516 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:20:59 smithi005 ceph-mon[29523]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-22T05:20:59.516 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:20:59 smithi005 podman[49318]: 2024-03-22 05:20:59.140520985 +0000 UTC m=+0.018577163 image pull ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4 quay.io/prometheus/alertmanager:v0.23.0 2024-03-22T05:20:59.516 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:20:59 smithi005 podman[49318]: 2024-03-22 05:20:59.257102542 +0000 UTC m=+0.135158719 volume create 175631a1742e331f5ceabda72ba773161ec1b42c76f0650816c7e61895203e63 2024-03-22T05:20:59.516 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:20:59 smithi005 podman[49318]: 2024-03-22 05:20:59.324080704 +0000 UTC m=+0.202136882 container create ee43e9b1c4278b9c5af3ae18dd8fc957a70592015277dee687bc39fbe5801362 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-22T05:20:59.516 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:59 smithi005 ceph-mon[24793]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-03-22T05:20:59.516 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:59 smithi005 ceph-mon[24793]: osdmap e68: 8 total, 8 up, 8 in 2024-03-22T05:20:59.516 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:59 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/971354302' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-22T05:20:59.517 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:20:59 smithi005 ceph-mon[24793]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-22T05:20:59.757 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:59 smithi186 ceph-mon[25720]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-03-22T05:20:59.757 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:59 smithi186 ceph-mon[25720]: osdmap e68: 8 total, 8 up, 8 in 2024-03-22T05:20:59.757 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:59 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/971354302' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-22T05:20:59.757 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:20:59 smithi186 ceph-mon[25720]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-22T05:20:59.952 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:20:59 smithi005 podman[49318]: 2024-03-22 05:20:59.601879988 +0000 UTC m=+0.479936166 container init ee43e9b1c4278b9c5af3ae18dd8fc957a70592015277dee687bc39fbe5801362 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-22T05:20:59.952 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:20:59 smithi005 podman[49318]: 2024-03-22 05:20:59.604468047 +0000 UTC m=+0.482524229 container start ee43e9b1c4278b9c5af3ae18dd8fc957a70592015277dee687bc39fbe5801362 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-22T05:20:59.953 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:20:59 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[49335]: level=info ts=2024-03-22T05:20:59.614Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-03-22T05:20:59.953 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:20:59 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[49335]: level=info ts=2024-03-22T05:20:59.614Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-03-22T05:20:59.953 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:20:59 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[49335]: level=info ts=2024-03-22T05:20:59.615Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.5 port=9094 2024-03-22T05:20:59.953 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:20:59 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[49335]: level=info ts=2024-03-22T05:20:59.617Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-22T05:20:59.953 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:20:59 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[49335]: level=info ts=2024-03-22T05:20:59.642Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-22T05:20:59.953 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:20:59 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[49335]: level=info ts=2024-03-22T05:20:59.642Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-22T05:20:59.953 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:20:59 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[49335]: level=info ts=2024-03-22T05:20:59.644Z caller=main.go:518 msg=Listening address=:9093 2024-03-22T05:20:59.953 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:20:59 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[49335]: level=info ts=2024-03-22T05:20:59.645Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-03-22T05:20:59.953 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:20:59 smithi005 bash[49318]: ee43e9b1c4278b9c5af3ae18dd8fc957a70592015277dee687bc39fbe5801362 2024-03-22T05:20:59.953 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:20:59 smithi005 systemd[1]: Started Ceph alertmanager.a for f59f36e2-e80a-11ee-95cd-87774f69a715. 2024-03-22T05:21:00.501 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-03-22T05:21:00.501 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "9fc716c2-c9cc-4938-be29-8ce3845a20e4", 2024-03-22T05:21:00.501 INFO:teuthology.orchestra.run.smithi005.stdout: "epoch": 1, 2024-03-22T05:21:00.501 INFO:teuthology.orchestra.run.smithi005.stdout: "predecessor_uuid": "1207cb3d-1f26-4c53-9152-ccd96f5df313", 2024-03-22T05:21:00.501 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_status": [], 2024-03-22T05:21:00.501 INFO:teuthology.orchestra.run.smithi005.stdout: "period_map": { 2024-03-22T05:21:00.502 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "9fc716c2-c9cc-4938-be29-8ce3845a20e4", 2024-03-22T05:21:00.502 INFO:teuthology.orchestra.run.smithi005.stdout: "zonegroups": [ 2024-03-22T05:21:00.502 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-03-22T05:21:00.502 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "4faa26b9-70eb-48ae-81f5-634fd700721a", 2024-03-22T05:21:00.502 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "default", 2024-03-22T05:21:00.502 INFO:teuthology.orchestra.run.smithi005.stdout: "api_name": "default", 2024-03-22T05:21:00.502 INFO:teuthology.orchestra.run.smithi005.stdout: "is_master": "true", 2024-03-22T05:21:00.502 INFO:teuthology.orchestra.run.smithi005.stdout: "endpoints": [], 2024-03-22T05:21:00.502 INFO:teuthology.orchestra.run.smithi005.stdout: "hostnames": [], 2024-03-22T05:21:00.502 INFO:teuthology.orchestra.run.smithi005.stdout: "hostnames_s3website": [], 2024-03-22T05:21:00.502 INFO:teuthology.orchestra.run.smithi005.stdout: "master_zone": "97eb5063-a634-49fc-801b-0370c4f7e8c9", 2024-03-22T05:21:00.502 INFO:teuthology.orchestra.run.smithi005.stdout: "zones": [ 2024-03-22T05:21:00.502 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-03-22T05:21:00.502 INFO:teuthology.orchestra.run.smithi005.stdout: "id": "97eb5063-a634-49fc-801b-0370c4f7e8c9", 2024-03-22T05:21:00.503 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "z", 2024-03-22T05:21:00.503 INFO:teuthology.orchestra.run.smithi005.stdout: "endpoints": [], 2024-03-22T05:21:00.503 INFO:teuthology.orchestra.run.smithi005.stdout: "log_meta": "false", 2024-03-22T05:21:00.503 INFO:teuthology.orchestra.run.smithi005.stdout: "log_data": "false", 2024-03-22T05:21:00.503 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket_index_max_shards": 11, 2024-03-22T05:21:00.503 INFO:teuthology.orchestra.run.smithi005.stdout: "read_only": "false", 2024-03-22T05:21:00.503 INFO:teuthology.orchestra.run.smithi005.stdout: "tier_type": "", 2024-03-22T05:21:00.503 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_from_all": "true", 2024-03-22T05:21:00.503 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_from": [], 2024-03-22T05:21:00.503 INFO:teuthology.orchestra.run.smithi005.stdout: "redirect_zone": "" 2024-03-22T05:21:00.503 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-03-22T05:21:00.503 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-03-22T05:21:00.503 INFO:teuthology.orchestra.run.smithi005.stdout: "placement_targets": [ 2024-03-22T05:21:00.503 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-03-22T05:21:00.503 INFO:teuthology.orchestra.run.smithi005.stdout: "name": "default-placement", 2024-03-22T05:21:00.504 INFO:teuthology.orchestra.run.smithi005.stdout: "tags": [], 2024-03-22T05:21:00.504 INFO:teuthology.orchestra.run.smithi005.stdout: "storage_classes": [ 2024-03-22T05:21:00.504 INFO:teuthology.orchestra.run.smithi005.stdout: "STANDARD" 2024-03-22T05:21:00.504 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-03-22T05:21:00.504 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-03-22T05:21:00.504 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-03-22T05:21:00.504 INFO:teuthology.orchestra.run.smithi005.stdout: "default_placement": "default-placement", 2024-03-22T05:21:00.504 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_id": "9bf819c1-7665-4d37-b4e2-f78c50dd5f3e", 2024-03-22T05:21:00.504 INFO:teuthology.orchestra.run.smithi005.stdout: "sync_policy": { 2024-03-22T05:21:00.504 INFO:teuthology.orchestra.run.smithi005.stdout: "groups": [] 2024-03-22T05:21:00.504 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-03-22T05:21:00.504 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-03-22T05:21:00.504 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-03-22T05:21:00.504 INFO:teuthology.orchestra.run.smithi005.stdout: "short_zone_ids": [ 2024-03-22T05:21:00.504 INFO:teuthology.orchestra.run.smithi005.stdout: { 2024-03-22T05:21:00.505 INFO:teuthology.orchestra.run.smithi005.stdout: "key": "97eb5063-a634-49fc-801b-0370c4f7e8c9", 2024-03-22T05:21:00.505 INFO:teuthology.orchestra.run.smithi005.stdout: "val": 3660227156 2024-03-22T05:21:00.505 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-03-22T05:21:00.505 INFO:teuthology.orchestra.run.smithi005.stdout: ] 2024-03-22T05:21:00.505 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:21:00.505 INFO:teuthology.orchestra.run.smithi005.stdout: "master_zonegroup": "4faa26b9-70eb-48ae-81f5-634fd700721a", 2024-03-22T05:21:00.505 INFO:teuthology.orchestra.run.smithi005.stdout: "master_zone": "97eb5063-a634-49fc-801b-0370c4f7e8c9", 2024-03-22T05:21:00.505 INFO:teuthology.orchestra.run.smithi005.stdout: "period_config": { 2024-03-22T05:21:00.505 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket_quota": { 2024-03-22T05:21:00.505 INFO:teuthology.orchestra.run.smithi005.stdout: "enabled": false, 2024-03-22T05:21:00.505 INFO:teuthology.orchestra.run.smithi005.stdout: "check_on_raw": false, 2024-03-22T05:21:00.505 INFO:teuthology.orchestra.run.smithi005.stdout: "max_size": -1, 2024-03-22T05:21:00.505 INFO:teuthology.orchestra.run.smithi005.stdout: "max_size_kb": 0, 2024-03-22T05:21:00.505 INFO:teuthology.orchestra.run.smithi005.stdout: "max_objects": -1 2024-03-22T05:21:00.505 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:21:00.506 INFO:teuthology.orchestra.run.smithi005.stdout: "user_quota": { 2024-03-22T05:21:00.506 INFO:teuthology.orchestra.run.smithi005.stdout: "enabled": false, 2024-03-22T05:21:00.506 INFO:teuthology.orchestra.run.smithi005.stdout: "check_on_raw": false, 2024-03-22T05:21:00.506 INFO:teuthology.orchestra.run.smithi005.stdout: "max_size": -1, 2024-03-22T05:21:00.506 INFO:teuthology.orchestra.run.smithi005.stdout: "max_size_kb": 0, 2024-03-22T05:21:00.506 INFO:teuthology.orchestra.run.smithi005.stdout: "max_objects": -1 2024-03-22T05:21:00.506 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:21:00.506 INFO:teuthology.orchestra.run.smithi005.stdout: "user_ratelimit": { 2024-03-22T05:21:00.506 INFO:teuthology.orchestra.run.smithi005.stdout: "max_read_ops": 0, 2024-03-22T05:21:00.507 INFO:teuthology.orchestra.run.smithi005.stdout: "max_write_ops": 0, 2024-03-22T05:21:00.507 INFO:teuthology.orchestra.run.smithi005.stdout: "max_read_bytes": 0, 2024-03-22T05:21:00.507 INFO:teuthology.orchestra.run.smithi005.stdout: "max_write_bytes": 0, 2024-03-22T05:21:00.507 INFO:teuthology.orchestra.run.smithi005.stdout: "enabled": false 2024-03-22T05:21:00.507 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:21:00.507 INFO:teuthology.orchestra.run.smithi005.stdout: "bucket_ratelimit": { 2024-03-22T05:21:00.507 INFO:teuthology.orchestra.run.smithi005.stdout: "max_read_ops": 0, 2024-03-22T05:21:00.507 INFO:teuthology.orchestra.run.smithi005.stdout: "max_write_ops": 0, 2024-03-22T05:21:00.507 INFO:teuthology.orchestra.run.smithi005.stdout: "max_read_bytes": 0, 2024-03-22T05:21:00.507 INFO:teuthology.orchestra.run.smithi005.stdout: "max_write_bytes": 0, 2024-03-22T05:21:00.507 INFO:teuthology.orchestra.run.smithi005.stdout: "enabled": false 2024-03-22T05:21:00.507 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:21:00.507 INFO:teuthology.orchestra.run.smithi005.stdout: "anonymous_ratelimit": { 2024-03-22T05:21:00.507 INFO:teuthology.orchestra.run.smithi005.stdout: "max_read_ops": 0, 2024-03-22T05:21:00.508 INFO:teuthology.orchestra.run.smithi005.stdout: "max_write_ops": 0, 2024-03-22T05:21:00.508 INFO:teuthology.orchestra.run.smithi005.stdout: "max_read_bytes": 0, 2024-03-22T05:21:00.508 INFO:teuthology.orchestra.run.smithi005.stdout: "max_write_bytes": 0, 2024-03-22T05:21:00.508 INFO:teuthology.orchestra.run.smithi005.stdout: "enabled": false 2024-03-22T05:21:00.508 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-03-22T05:21:00.508 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:21:00.508 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_id": "9bf819c1-7665-4d37-b4e2-f78c50dd5f3e", 2024-03-22T05:21:00.508 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_name": "r", 2024-03-22T05:21:00.508 INFO:teuthology.orchestra.run.smithi005.stdout: "realm_epoch": 2 2024-03-22T05:21:00.508 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-03-22T05:21:00.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:00 smithi005 ceph-mon[29523]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-22T05:21:00.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:00 smithi005 ceph-mon[29523]: osdmap e69: 8 total, 8 up, 8 in 2024-03-22T05:21:00.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:00 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/971354302' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-03-22T05:21:00.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:00 smithi005 ceph-mon[29523]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-03-22T05:21:00.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:00 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:00.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:00 smithi005 ceph-mon[29523]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-22T05:21:00.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:00 smithi005 ceph-mon[29523]: Reconfiguring daemon prometheus.a on smithi186 2024-03-22T05:21:00.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:00 smithi005 ceph-mon[29523]: pgmap v59: 129 pgs: 129 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-03-22T05:21:00.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:00 smithi005 ceph-mon[24793]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-22T05:21:00.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:00 smithi005 ceph-mon[24793]: osdmap e69: 8 total, 8 up, 8 in 2024-03-22T05:21:00.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:00 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/971354302' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-03-22T05:21:00.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:00 smithi005 ceph-mon[24793]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-03-22T05:21:00.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:00 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:00.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:00 smithi005 ceph-mon[24793]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-22T05:21:00.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:00 smithi005 ceph-mon[24793]: Reconfiguring daemon prometheus.a on smithi186 2024-03-22T05:21:00.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:00 smithi005 ceph-mon[24793]: pgmap v59: 129 pgs: 129 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-03-22T05:21:00.758 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:00 smithi186 ceph-mon[25720]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-22T05:21:00.758 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:00 smithi186 ceph-mon[25720]: osdmap e69: 8 total, 8 up, 8 in 2024-03-22T05:21:00.758 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:00 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/971354302' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-03-22T05:21:00.759 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:00 smithi186 ceph-mon[25720]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-03-22T05:21:00.759 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:00 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:00.759 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:00 smithi186 ceph-mon[25720]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-22T05:21:00.759 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:00 smithi186 ceph-mon[25720]: Reconfiguring daemon prometheus.a on smithi186 2024-03-22T05:21:00.759 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:00 smithi186 ceph-mon[25720]: pgmap v59: 129 pgs: 129 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-03-22T05:21:01.233 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000' 2024-03-22T05:21:01.554 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:01 smithi005 ceph-mon[29523]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-03-22T05:21:01.554 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:01 smithi005 ceph-mon[29523]: osdmap e70: 8 total, 8 up, 8 in 2024-03-22T05:21:01.554 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:01 smithi005 ceph-mon[24793]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-03-22T05:21:01.554 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:01 smithi005 ceph-mon[24793]: osdmap e70: 8 total, 8 up, 8 in 2024-03-22T05:21:01.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:01 smithi186 ceph-mon[25720]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-03-22T05:21:01.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:01 smithi186 ceph-mon[25720]: osdmap e70: 8 total, 8 up, 8 in 2024-03-22T05:21:01.866 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:21:01 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[49335]: level=info ts=2024-03-22T05:21:01.617Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000145964s 2024-03-22T05:21:02.445 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:02 smithi005 ceph-mon[24793]: pgmap v61: 129 pgs: 129 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 224 B/s rd, 449 B/s wr, 1 op/s 2024-03-22T05:21:02.702 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:21:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:21:02] "GET /metrics HTTP/1.1" 200 196333 "" "Prometheus/2.33.4" 2024-03-22T05:21:02.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:02 smithi005 ceph-mon[29523]: pgmap v61: 129 pgs: 129 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 224 B/s rd, 449 B/s wr, 1 op/s 2024-03-22T05:21:02.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:02 smithi186 ceph-mon[25720]: pgmap v61: 129 pgs: 129 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 224 B/s rd, 449 B/s wr, 1 op/s 2024-03-22T05:21:03.071 INFO:teuthology.orchestra.run.smithi005.stdout:Scheduled rgw.foo update... 2024-03-22T05:21:03.192 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:21:03 smithi186 systemd[1]: Stopping Ceph prometheus.a for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:21:03.192 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:21:03 smithi186 bash[42474]: Error: no container with name or ID "ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus.a" found: no such container 2024-03-22T05:21:03.486 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:21:03 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:21:03] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-22T05:21:03.486 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:21:03 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[40776]: ts=2024-03-22T05:21:03.221Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-22T05:21:03.487 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:21:03 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[40776]: ts=2024-03-22T05:21:03.221Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-03-22T05:21:03.487 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:21:03 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[40776]: ts=2024-03-22T05:21:03.221Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-03-22T05:21:03.487 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:21:03 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[40776]: ts=2024-03-22T05:21:03.221Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-03-22T05:21:03.487 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:21:03 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[40776]: ts=2024-03-22T05:21:03.221Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-03-22T05:21:03.487 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:21:03 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[40776]: ts=2024-03-22T05:21:03.221Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-03-22T05:21:03.487 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:21:03 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[40776]: ts=2024-03-22T05:21:03.221Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-22T05:21:03.487 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:21:03 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[40776]: ts=2024-03-22T05:21:03.221Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-03-22T05:21:03.487 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:21:03 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[40776]: ts=2024-03-22T05:21:03.221Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-03-22T05:21:03.487 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:21:03 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[40776]: ts=2024-03-22T05:21:03.222Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-03-22T05:21:03.487 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:21:03 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[40776]: ts=2024-03-22T05:21:03.222Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-03-22T05:21:03.487 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:21:03 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[40776]: ts=2024-03-22T05:21:03.222Z caller=main.go:1066 level=info msg="See you next time!" 2024-03-22T05:21:03.487 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:21:03 smithi186 podman[42487]: 2024-03-22 05:21:03.231692353 +0000 UTC m=+0.135707648 container died 1f0b532ae06c8797648881a188550e79e0d6311c97b1834da3c3eac62bce3e9e (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-22T05:21:03.669 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph osd pool create foo' 2024-03-22T05:21:03.793 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:21:03 smithi186 podman[42487]: 2024-03-22 05:21:03.669529462 +0000 UTC m=+0.573544767 container cleanup 1f0b532ae06c8797648881a188550e79e0d6311c97b1834da3c3eac62bce3e9e (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-22T05:21:03.794 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:21:03 smithi186 bash[42487]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a 2024-03-22T05:21:04.166 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:04 smithi186 ceph-mon[25720]: from='client.24592 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:21:04.166 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:04 smithi186 ceph-mon[25720]: Saving service rgw.foo spec with placement count:2 2024-03-22T05:21:04.167 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:04 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:04.167 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:21:03 smithi186 podman[42501]: 2024-03-22 05:21:03.887473784 +0000 UTC m=+0.658361556 container remove 1f0b532ae06c8797648881a188550e79e0d6311c97b1834da3c3eac62bce3e9e (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-22T05:21:04.167 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:21:03 smithi186 bash[42515]: Error: no container with name or ID "ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus.a" found: no such container 2024-03-22T05:21:04.167 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:21:03 smithi186 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-03-22T05:21:04.167 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:21:03 smithi186 systemd[1]: Stopped Ceph prometheus.a for f59f36e2-e80a-11ee-95cd-87774f69a715. 2024-03-22T05:21:04.167 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:21:03 smithi186 systemd[1]: Starting Ceph prometheus.a for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:21:04.271 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:04 smithi005 ceph-mon[24793]: from='client.24592 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:21:04.271 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:04 smithi005 ceph-mon[24793]: Saving service rgw.foo spec with placement count:2 2024-03-22T05:21:04.271 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:04 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:04.272 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:04 smithi005 ceph-mon[29523]: from='client.24592 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:21:04.272 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:04 smithi005 ceph-mon[29523]: Saving service rgw.foo spec with placement count:2 2024-03-22T05:21:04.272 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:04 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:04.434 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:21:04 smithi186 podman[42581]: 2024-03-22 05:21:04.071992212 +0000 UTC m=+0.020191631 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-03-22T05:21:04.434 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:21:04 smithi186 podman[42581]: 2024-03-22 05:21:04.171925092 +0000 UTC m=+0.120124498 container create f520626945ac93d954db1d7a0d807945f01fbffaf6947b3a66590fcc2f0ee9e6 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-22T05:21:04.434 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:21:04 smithi186 podman[42581]: 2024-03-22 05:21:04.407665469 +0000 UTC m=+0.355864889 container init f520626945ac93d954db1d7a0d807945f01fbffaf6947b3a66590fcc2f0ee9e6 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-22T05:21:04.434 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:21:04 smithi186 podman[42581]: 2024-03-22 05:21:04.410238014 +0000 UTC m=+0.358437421 container start f520626945ac93d954db1d7a0d807945f01fbffaf6947b3a66590fcc2f0ee9e6 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-22T05:21:04.794 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:21:04 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[42595]: ts=2024-03-22T05:21:04.434Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-03-22T05:21:04.794 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:21:04 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[42595]: ts=2024-03-22T05:21:04.434Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-03-22T05:21:04.794 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:21:04 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[42595]: ts=2024-03-22T05:21:04.434Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-03-22T05:21:04.794 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:21:04 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[42595]: ts=2024-03-22T05:21:04.434Z caller=main.go:518 level=info host_details="(Linux 5.14.0-430.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024 x86_64 smithi186 (none))" 2024-03-22T05:21:04.794 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:21:04 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[42595]: ts=2024-03-22T05:21:04.434Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-22T05:21:04.794 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:21:04 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[42595]: ts=2024-03-22T05:21:04.434Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-22T05:21:04.795 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:21:04 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[42595]: ts=2024-03-22T05:21:04.436Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-03-22T05:21:04.795 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:21:04 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[42595]: ts=2024-03-22T05:21:04.436Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-03-22T05:21:04.795 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:21:04 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[42595]: ts=2024-03-22T05:21:04.438Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-03-22T05:21:04.795 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:21:04 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[42595]: ts=2024-03-22T05:21:04.440Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-22T05:21:04.795 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:21:04 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[42595]: ts=2024-03-22T05:21:04.440Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.388µs 2024-03-22T05:21:04.795 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:21:04 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[42595]: ts=2024-03-22T05:21:04.440Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-22T05:21:04.795 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:21:04 smithi186 bash[42581]: f520626945ac93d954db1d7a0d807945f01fbffaf6947b3a66590fcc2f0ee9e6 2024-03-22T05:21:04.796 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:21:04 smithi186 systemd[1]: Started Ceph prometheus.a for f59f36e2-e80a-11ee-95cd-87774f69a715. 2024-03-22T05:21:04.796 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:21:04 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[42595]: ts=2024-03-22T05:21:04.656Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-03-22T05:21:04.796 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:21:04 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[42595]: ts=2024-03-22T05:21:04.657Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-03-22T05:21:04.796 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:21:04 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[42595]: ts=2024-03-22T05:21:04.657Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=46.702µs wal_replay_duration=216.701983ms total_replay_duration=216.769875ms 2024-03-22T05:21:04.796 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:21:04 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[42595]: ts=2024-03-22T05:21:04.658Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-03-22T05:21:04.796 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:21:04 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[42595]: ts=2024-03-22T05:21:04.658Z caller=main.go:947 level=info msg="TSDB started" 2024-03-22T05:21:04.796 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:21:04 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[42595]: ts=2024-03-22T05:21:04.658Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-22T05:21:04.797 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:21:04 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[42595]: ts=2024-03-22T05:21:04.675Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=16.399574ms db_storage=643ns remote_storage=1.66µs web_handler=466ns query_engine=694ns scrape=341.837µs scrape_sd=22.634µs notify=19.225µs notify_sd=8.849µs rules=15.80532ms 2024-03-22T05:21:04.797 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:21:04 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[42595]: ts=2024-03-22T05:21:04.675Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-03-22T05:21:05.316 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:05 smithi005 ceph-mon[24793]: pgmap v62: 129 pgs: 129 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-22T05:21:05.316 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:05 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:05.316 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:05 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-22T05:21:05.316 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:05 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.5:9093"}]: dispatch 2024-03-22T05:21:05.317 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:05 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:05.317 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:05 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-22T05:21:05.317 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:05 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.186:9095"}]: dispatch 2024-03-22T05:21:05.317 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:05 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:05.317 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:05 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-22T05:21:05.318 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:05 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.186:3000"}]: dispatch 2024-03-22T05:21:05.318 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:05 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:05.318 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:05 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:21:05.318 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:05 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:21:05.318 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:05 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:21:05.319 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:05 smithi005 ceph-mon[29523]: pgmap v62: 129 pgs: 129 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-22T05:21:05.319 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:05 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:05.319 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:05 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-22T05:21:05.319 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:05 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.5:9093"}]: dispatch 2024-03-22T05:21:05.319 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:05 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:05.320 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:05 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-22T05:21:05.320 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:05 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.186:9095"}]: dispatch 2024-03-22T05:21:05.320 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:05 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:05.320 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:05 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-22T05:21:05.320 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:05 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.186:3000"}]: dispatch 2024-03-22T05:21:05.320 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:05 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:05.321 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:05 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:21:05.321 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:05 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:21:05.321 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:05 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:21:05.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:05 smithi186 ceph-mon[25720]: pgmap v62: 129 pgs: 129 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-22T05:21:05.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:05 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:05.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:05 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-22T05:21:05.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:05 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.5:9093"}]: dispatch 2024-03-22T05:21:05.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:05 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:05.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:05 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-22T05:21:05.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:05 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.186:9095"}]: dispatch 2024-03-22T05:21:05.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:05 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:05.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:05 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-22T05:21:05.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:05 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.186:3000"}]: dispatch 2024-03-22T05:21:05.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:05 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:05.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:05 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:21:05.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:05 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:21:05.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:05 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:21:06.172 INFO:teuthology.orchestra.run.smithi005.stderr:pool 'foo' created 2024-03-22T05:21:06.380 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:06 smithi186 ceph-mon[25720]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-22T05:21:06.380 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:06 smithi186 ceph-mon[25720]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.5:9093"}]: dispatch 2024-03-22T05:21:06.380 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:06 smithi186 ceph-mon[25720]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-22T05:21:06.380 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:06 smithi186 ceph-mon[25720]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.186:9095"}]: dispatch 2024-03-22T05:21:06.381 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:06 smithi186 ceph-mon[25720]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-22T05:21:06.381 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:06 smithi186 ceph-mon[25720]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.186:3000"}]: dispatch 2024-03-22T05:21:06.411 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:06 smithi005 ceph-mon[24793]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-22T05:21:06.411 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:06 smithi005 ceph-mon[24793]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.5:9093"}]: dispatch 2024-03-22T05:21:06.411 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:06 smithi005 ceph-mon[24793]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-22T05:21:06.412 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:06 smithi005 ceph-mon[24793]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.186:9095"}]: dispatch 2024-03-22T05:21:06.412 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:06 smithi005 ceph-mon[24793]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-22T05:21:06.412 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:06 smithi005 ceph-mon[24793]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.186:3000"}]: dispatch 2024-03-22T05:21:06.412 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:06 smithi005 ceph-mon[29523]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-22T05:21:06.412 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:06 smithi005 ceph-mon[29523]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.5:9093"}]: dispatch 2024-03-22T05:21:06.412 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:06 smithi005 ceph-mon[29523]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-22T05:21:06.412 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:06 smithi005 ceph-mon[29523]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.186:9095"}]: dispatch 2024-03-22T05:21:06.412 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:06 smithi005 ceph-mon[29523]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-22T05:21:06.413 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:06 smithi005 ceph-mon[29523]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.186:3000"}]: dispatch 2024-03-22T05:21:07.035 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'rbd pool init foo' 2024-03-22T05:21:07.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:07 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/2470372353' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-03-22T05:21:07.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:07 smithi186 ceph-mon[25720]: pgmap v63: 129 pgs: 129 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-22T05:21:07.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:07 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/2470372353' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-03-22T05:21:07.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:07 smithi186 ceph-mon[25720]: osdmap e71: 8 total, 8 up, 8 in 2024-03-22T05:21:07.387 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:07 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/2470372353' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-03-22T05:21:07.388 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:07 smithi005 ceph-mon[24793]: pgmap v63: 129 pgs: 129 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-22T05:21:07.388 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:07 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/2470372353' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-03-22T05:21:07.390 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:07 smithi005 ceph-mon[24793]: osdmap e71: 8 total, 8 up, 8 in 2024-03-22T05:21:07.390 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:07 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/2470372353' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-03-22T05:21:07.390 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:07 smithi005 ceph-mon[29523]: pgmap v63: 129 pgs: 129 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-22T05:21:07.391 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:07 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/2470372353' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-03-22T05:21:07.391 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:07 smithi005 ceph-mon[29523]: osdmap e71: 8 total, 8 up, 8 in 2024-03-22T05:21:08.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:08 smithi005 ceph-mon[24793]: osdmap e72: 8 total, 8 up, 8 in 2024-03-22T05:21:08.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:08 smithi005 ceph-mon[24793]: pgmap v66: 161 pgs: 32 unknown, 129 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 950 B/s rd, 950 B/s wr, 2 op/s 2024-03-22T05:21:08.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:08 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:08.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:08 smithi005 ceph-mon[29523]: osdmap e72: 8 total, 8 up, 8 in 2024-03-22T05:21:08.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:08 smithi005 ceph-mon[29523]: pgmap v66: 161 pgs: 32 unknown, 129 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 950 B/s rd, 950 B/s wr, 2 op/s 2024-03-22T05:21:08.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:08 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:08.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:08 smithi186 ceph-mon[25720]: osdmap e72: 8 total, 8 up, 8 in 2024-03-22T05:21:08.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:08 smithi186 ceph-mon[25720]: pgmap v66: 161 pgs: 32 unknown, 129 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 950 B/s rd, 950 B/s wr, 2 op/s 2024-03-22T05:21:08.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:08 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:09.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:09 smithi005 ceph-mon[24793]: osdmap e73: 8 total, 8 up, 8 in 2024-03-22T05:21:09.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:09 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/3326073944' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-03-22T05:21:09.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:09 smithi005 ceph-mon[24793]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-03-22T05:21:09.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:09 smithi005 ceph-mon[29523]: osdmap e73: 8 total, 8 up, 8 in 2024-03-22T05:21:09.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:09 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/3326073944' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-03-22T05:21:09.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:09 smithi005 ceph-mon[29523]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-03-22T05:21:09.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:09 smithi186 ceph-mon[25720]: osdmap e73: 8 total, 8 up, 8 in 2024-03-22T05:21:09.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:09 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/3326073944' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-03-22T05:21:09.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:09 smithi186 ceph-mon[25720]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-03-22T05:21:09.878 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:21:09 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[49335]: level=info ts=2024-03-22T05:21:09.620Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.003392211s 2024-03-22T05:21:10.423 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:10 smithi005 ceph-mon[24793]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-03-22T05:21:10.423 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:10 smithi005 ceph-mon[24793]: osdmap e74: 8 total, 8 up, 8 in 2024-03-22T05:21:10.423 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:10 smithi005 ceph-mon[24793]: pgmap v69: 161 pgs: 161 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:21:10.423 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:10 smithi005 ceph-mon[29523]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-03-22T05:21:10.424 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:10 smithi005 ceph-mon[29523]: osdmap e74: 8 total, 8 up, 8 in 2024-03-22T05:21:10.424 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:10 smithi005 ceph-mon[29523]: pgmap v69: 161 pgs: 161 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:21:10.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:10 smithi186 ceph-mon[25720]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-03-22T05:21:10.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:10 smithi186 ceph-mon[25720]: osdmap e74: 8 total, 8 up, 8 in 2024-03-22T05:21:10.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:10 smithi186 ceph-mon[25720]: pgmap v69: 161 pgs: 161 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:21:11.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:11 smithi005 ceph-mon[24793]: osdmap e75: 8 total, 8 up, 8 in 2024-03-22T05:21:11.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:11 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:11.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:11 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:11.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:11 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:11.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:11 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:11.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:11 smithi005 ceph-mon[24793]: Saving service rgw.foo spec with placement count:2 2024-03-22T05:21:11.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:11 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:11.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:11 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi005.exdhqu", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-22T05:21:11.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:11 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi005.exdhqu", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-22T05:21:11.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:11 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:11.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:11 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:21:11.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:11 smithi005 ceph-mon[24793]: Deploying daemon rgw.foo.smithi005.exdhqu on smithi005 2024-03-22T05:21:11.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:11 smithi005 ceph-mon[29523]: osdmap e75: 8 total, 8 up, 8 in 2024-03-22T05:21:11.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:11 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:11.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:11 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:11.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:11 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:11.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:11 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:11.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:11 smithi005 ceph-mon[29523]: Saving service rgw.foo spec with placement count:2 2024-03-22T05:21:11.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:11 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:11.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:11 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi005.exdhqu", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-22T05:21:11.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:11 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi005.exdhqu", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-22T05:21:11.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:11 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:11.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:11 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:21:11.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:11 smithi005 ceph-mon[29523]: Deploying daemon rgw.foo.smithi005.exdhqu on smithi005 2024-03-22T05:21:11.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:11 smithi186 ceph-mon[25720]: osdmap e75: 8 total, 8 up, 8 in 2024-03-22T05:21:11.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:11 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:11.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:11 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:11.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:11 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:11.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:11 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:11.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:11 smithi186 ceph-mon[25720]: Saving service rgw.foo spec with placement count:2 2024-03-22T05:21:11.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:11 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:11.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:11 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi005.exdhqu", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-22T05:21:11.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:11 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi005.exdhqu", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-22T05:21:11.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:11 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:11.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:11 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:21:11.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:11 smithi186 ceph-mon[25720]: Deploying daemon rgw.foo.smithi005.exdhqu on smithi005 2024-03-22T05:21:11.976 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph orch apply iscsi foo u p' 2024-03-22T05:21:12.407 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:12 smithi005 ceph-mon[24793]: osdmap e76: 8 total, 8 up, 8 in 2024-03-22T05:21:12.407 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:12 smithi005 ceph-mon[24793]: pgmap v72: 161 pgs: 161 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:21:12.408 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:12 smithi005 ceph-mon[29523]: osdmap e76: 8 total, 8 up, 8 in 2024-03-22T05:21:12.408 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:12 smithi005 ceph-mon[29523]: pgmap v72: 161 pgs: 161 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:21:12.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:12 smithi186 ceph-mon[25720]: osdmap e76: 8 total, 8 up, 8 in 2024-03-22T05:21:12.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:12 smithi186 ceph-mon[25720]: pgmap v72: 161 pgs: 161 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:21:12.702 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:21:12 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:21:12] "GET /metrics HTTP/1.1" 200 196333 "" "Prometheus/2.33.4" 2024-03-22T05:21:13.543 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:21:13 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:21:13] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-22T05:21:14.282 INFO:teuthology.orchestra.run.smithi005.stdout:Scheduled iscsi.foo update... 2024-03-22T05:21:14.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:14 smithi005 ceph-mon[24793]: pgmap v73: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 352 B/s wr, 0 op/s 2024-03-22T05:21:14.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:14 smithi005 ceph-mon[29523]: pgmap v73: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 352 B/s wr, 0 op/s 2024-03-22T05:21:14.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:14 smithi186 ceph-mon[25720]: pgmap v73: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 352 B/s wr, 0 op/s 2024-03-22T05:21:14.848 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'sleep 180' 2024-03-22T05:21:15.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:15 smithi005 ceph-mon[24793]: from='client.24610 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:21:15.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:15 smithi005 ceph-mon[24793]: Saving service iscsi.foo spec with placement count:1 2024-03-22T05:21:15.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:15 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:15.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:15 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:15.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:15 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi186.mcgodk", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-22T05:21:15.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:15 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi186.mcgodk", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-22T05:21:15.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:15 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:15.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:15 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:21:15.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:15 smithi005 ceph-mon[29523]: from='client.24610 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:21:15.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:15 smithi005 ceph-mon[29523]: Saving service iscsi.foo spec with placement count:1 2024-03-22T05:21:15.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:15 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:15.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:15 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:15.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:15 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi186.mcgodk", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-22T05:21:15.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:15 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi186.mcgodk", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-22T05:21:15.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:15 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:15.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:15 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:21:15.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:15 smithi186 ceph-mon[25720]: from='client.24610 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:21:15.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:15 smithi186 ceph-mon[25720]: Saving service iscsi.foo spec with placement count:1 2024-03-22T05:21:15.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:15 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:15.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:15 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:15.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:15 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi186.mcgodk", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-22T05:21:15.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:15 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi186.mcgodk", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-22T05:21:15.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:15 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:15.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:15 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:21:16.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:16 smithi186 ceph-mon[25720]: Deploying daemon rgw.foo.smithi186.mcgodk on smithi186 2024-03-22T05:21:16.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:16 smithi186 ceph-mon[25720]: pgmap v74: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 300 B/s wr, 0 op/s 2024-03-22T05:21:16.548 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:16 smithi005 ceph-mon[24793]: Deploying daemon rgw.foo.smithi186.mcgodk on smithi186 2024-03-22T05:21:16.548 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:16 smithi005 ceph-mon[24793]: pgmap v74: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 300 B/s wr, 0 op/s 2024-03-22T05:21:16.548 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:16 smithi005 ceph-mon[29523]: Deploying daemon rgw.foo.smithi186.mcgodk on smithi186 2024-03-22T05:21:16.548 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:16 smithi005 ceph-mon[29523]: pgmap v74: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 300 B/s wr, 0 op/s 2024-03-22T05:21:19.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:19 smithi005 ceph-mon[29523]: pgmap v75: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-03-22T05:21:19.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:19 smithi005 ceph-mon[29523]: osdmap e77: 8 total, 8 up, 8 in 2024-03-22T05:21:19.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:19 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:19.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:19 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:21:19.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:19 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:21:19.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:19 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:21:19.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:19 smithi005 ceph-mon[24793]: pgmap v75: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-03-22T05:21:19.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:19 smithi005 ceph-mon[24793]: osdmap e77: 8 total, 8 up, 8 in 2024-03-22T05:21:19.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:19 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:19.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:19 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:21:19.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:19 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:21:19.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:19 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:21:19.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:19 smithi186 ceph-mon[25720]: pgmap v75: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-03-22T05:21:19.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:19 smithi186 ceph-mon[25720]: osdmap e77: 8 total, 8 up, 8 in 2024-03-22T05:21:19.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:19 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:19.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:19 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:21:19.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:19 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:21:19.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:19 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:21:21.254 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:21 smithi005 ceph-mon[24793]: pgmap v77: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 163 KiB/s rd, 4.1 KiB/s wr, 296 op/s 2024-03-22T05:21:21.255 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:21 smithi005 ceph-mon[29523]: pgmap v77: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 163 KiB/s rd, 4.1 KiB/s wr, 296 op/s 2024-03-22T05:21:21.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:21 smithi186 ceph-mon[25720]: pgmap v77: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 163 KiB/s rd, 4.1 KiB/s wr, 296 op/s 2024-03-22T05:21:22.702 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:21:22 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:21:22] "GET /metrics HTTP/1.1" 200 197808 "" "Prometheus/2.33.4" 2024-03-22T05:21:22.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:22 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:22.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:22 smithi005 ceph-mon[24793]: pgmap v78: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 143 KiB/s rd, 3.6 KiB/s wr, 261 op/s 2024-03-22T05:21:22.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:22 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:22.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:22 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:22.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:22 smithi005 ceph-mon[29523]: pgmap v78: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 143 KiB/s rd, 3.6 KiB/s wr, 261 op/s 2024-03-22T05:21:22.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:22 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:22.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:22 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:22.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:22 smithi186 ceph-mon[25720]: pgmap v78: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 143 KiB/s rd, 3.6 KiB/s wr, 261 op/s 2024-03-22T05:21:22.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:22 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:23.543 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:21:23 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:21:23] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-22T05:21:23.678 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:23 smithi005 ceph-mon[24793]: Checking dashboard <-> RGW credentials 2024-03-22T05:21:23.678 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:23 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:23.678 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:23 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:23.678 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:23 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:23.678 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:23 smithi005 ceph-mon[24793]: Checking pool "foo" exists for service iscsi.foo 2024-03-22T05:21:23.678 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:23 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:21:23.679 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:23 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-22T05:21:23.679 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:23 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:21:23.679 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:23 smithi005 ceph-mon[24793]: Deploying daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:21:23.679 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:23 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:23.679 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:23 smithi005 ceph-mon[29523]: Checking dashboard <-> RGW credentials 2024-03-22T05:21:23.679 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:23 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:23.679 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:23 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:23.679 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:23 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:23.679 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:23 smithi005 ceph-mon[29523]: Checking pool "foo" exists for service iscsi.foo 2024-03-22T05:21:23.680 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:23 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:21:23.680 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:23 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-22T05:21:23.680 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:23 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:21:23.680 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:23 smithi005 ceph-mon[29523]: Deploying daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:21:23.680 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:23 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:24.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:23 smithi186 ceph-mon[25720]: Checking dashboard <-> RGW credentials 2024-03-22T05:21:24.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:23 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:24.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:23 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:24.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:23 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:24.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:23 smithi186 ceph-mon[25720]: Checking pool "foo" exists for service iscsi.foo 2024-03-22T05:21:24.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:23 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:21:24.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:23 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-22T05:21:24.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:23 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:21:24.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:23 smithi186 ceph-mon[25720]: Deploying daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:21:24.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:23 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:24.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:24 smithi005 ceph-mon[24793]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 187 KiB/s rd, 3.6 KiB/s wr, 329 op/s 2024-03-22T05:21:24.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:24 smithi005 ceph-mon[29523]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 187 KiB/s rd, 3.6 KiB/s wr, 329 op/s 2024-03-22T05:21:25.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:24 smithi186 ceph-mon[25720]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 187 KiB/s rd, 3.6 KiB/s wr, 329 op/s 2024-03-22T05:21:25.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:25 smithi005 ceph-mon[24793]: mgrmap e21: y(active, since 117s), standbys: x 2024-03-22T05:21:25.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:25 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:25.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:25 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:21:25.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:25 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:21:25.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:25 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:21:25.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:25 smithi005 ceph-mon[29523]: mgrmap e21: y(active, since 117s), standbys: x 2024-03-22T05:21:25.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:25 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:25.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:25 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:21:25.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:25 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:21:25.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:25 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:21:25.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:25 smithi186 ceph-mon[25720]: mgrmap e21: y(active, since 117s), standbys: x 2024-03-22T05:21:25.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:25 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:25.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:25 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:21:25.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:25 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:21:25.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:25 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:21:26.838 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:26 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/1693411581' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-22T05:21:26.838 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:26 smithi005 ceph-mon[24793]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 187 KiB/s rd, 3.6 KiB/s wr, 329 op/s 2024-03-22T05:21:26.839 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:26 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/1307855459' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/757446544"}]: dispatch 2024-03-22T05:21:26.839 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:26 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/1693411581' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-22T05:21:26.839 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:26 smithi005 ceph-mon[29523]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 187 KiB/s rd, 3.6 KiB/s wr, 329 op/s 2024-03-22T05:21:26.839 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:26 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/1307855459' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/757446544"}]: dispatch 2024-03-22T05:21:26.924 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:26 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/1693411581' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-22T05:21:26.924 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:26 smithi186 ceph-mon[25720]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 187 KiB/s rd, 3.6 KiB/s wr, 329 op/s 2024-03-22T05:21:26.924 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:26 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/1307855459' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/757446544"}]: dispatch 2024-03-22T05:21:27.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:27 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/1307855459' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/757446544"}]': finished 2024-03-22T05:21:27.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:27 smithi005 ceph-mon[24793]: osdmap e78: 8 total, 8 up, 8 in 2024-03-22T05:21:27.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:27 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/2035649073' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6800/294522234"}]: dispatch 2024-03-22T05:21:27.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:27 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/1307855459' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/757446544"}]': finished 2024-03-22T05:21:27.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:27 smithi005 ceph-mon[29523]: osdmap e78: 8 total, 8 up, 8 in 2024-03-22T05:21:27.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:27 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/2035649073' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6800/294522234"}]: dispatch 2024-03-22T05:21:28.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:27 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/1307855459' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/757446544"}]': finished 2024-03-22T05:21:28.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:27 smithi186 ceph-mon[25720]: osdmap e78: 8 total, 8 up, 8 in 2024-03-22T05:21:28.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:27 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/2035649073' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6800/294522234"}]: dispatch 2024-03-22T05:21:28.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:28 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/2035649073' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6800/294522234"}]': finished 2024-03-22T05:21:28.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:28 smithi005 ceph-mon[24793]: osdmap e79: 8 total, 8 up, 8 in 2024-03-22T05:21:28.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:28 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/2545306374' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/2650789828"}]: dispatch 2024-03-22T05:21:28.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:28 smithi005 ceph-mon[24793]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 255 B/s wr, 85 op/s 2024-03-22T05:21:28.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:28 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [7, 4]}]: dispatch 2024-03-22T05:21:28.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:28 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-03-22T05:21:28.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:28 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 5]}]: dispatch 2024-03-22T05:21:28.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:28 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]: dispatch 2024-03-22T05:21:28.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:28 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-22T05:21:28.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:28 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-22T05:21:28.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:28 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:28.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:28 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/2035649073' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6800/294522234"}]': finished 2024-03-22T05:21:28.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:28 smithi005 ceph-mon[29523]: osdmap e79: 8 total, 8 up, 8 in 2024-03-22T05:21:28.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:28 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/2545306374' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/2650789828"}]: dispatch 2024-03-22T05:21:28.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:28 smithi005 ceph-mon[29523]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 255 B/s wr, 85 op/s 2024-03-22T05:21:28.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:28 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [7, 4]}]: dispatch 2024-03-22T05:21:28.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:28 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-03-22T05:21:28.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:28 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 5]}]: dispatch 2024-03-22T05:21:28.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:28 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]: dispatch 2024-03-22T05:21:28.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:28 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-22T05:21:28.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:28 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-22T05:21:28.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:28 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:29.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:28 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/2035649073' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6800/294522234"}]': finished 2024-03-22T05:21:29.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:28 smithi186 ceph-mon[25720]: osdmap e79: 8 total, 8 up, 8 in 2024-03-22T05:21:29.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:28 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/2545306374' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/2650789828"}]: dispatch 2024-03-22T05:21:29.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:28 smithi186 ceph-mon[25720]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 255 B/s wr, 85 op/s 2024-03-22T05:21:29.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:28 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [7, 4]}]: dispatch 2024-03-22T05:21:29.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:28 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-03-22T05:21:29.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:28 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 5]}]: dispatch 2024-03-22T05:21:29.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:28 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]: dispatch 2024-03-22T05:21:29.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:28 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-22T05:21:29.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:28 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-22T05:21:29.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:28 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:29.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:29 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/2545306374' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/2650789828"}]': finished 2024-03-22T05:21:29.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:29 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [7, 4]}]': finished 2024-03-22T05:21:29.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:29 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]': finished 2024-03-22T05:21:29.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:29 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 5]}]': finished 2024-03-22T05:21:29.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:29 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]': finished 2024-03-22T05:21:29.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:29 smithi005 ceph-mon[24793]: osdmap e80: 8 total, 8 up, 8 in 2024-03-22T05:21:29.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:29 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:29.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:29 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/698140406' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6801/2160302520"}]: dispatch 2024-03-22T05:21:29.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:29 smithi005 ceph-mon[24793]: from='client.? ' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6801/2160302520"}]: dispatch 2024-03-22T05:21:29.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:29 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:29.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:29 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:29.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:29 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-22T05:21:29.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:29 smithi005 ceph-mon[24793]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-22T05:21:29.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:29 smithi005 ceph-mon[24793]: Adding iSCSI gateway http://:@172.21.15.5:5000 to Dashboard 2024-03-22T05:21:29.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:29 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-22T05:21:29.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:29 smithi005 ceph-mon[24793]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-22T05:21:29.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:29 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:29.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:29 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi005"}]: dispatch 2024-03-22T05:21:29.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:29 smithi005 ceph-mon[24793]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi005"}]: dispatch 2024-03-22T05:21:29.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:29 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:29.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:29 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:21:29.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:29 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:21:29.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:29 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:21:29.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:29 smithi005 ceph-mon[24793]: Checking dashboard <-> RGW credentials 2024-03-22T05:21:29.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:29 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/2545306374' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/2650789828"}]': finished 2024-03-22T05:21:29.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:29 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [7, 4]}]': finished 2024-03-22T05:21:29.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:29 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]': finished 2024-03-22T05:21:29.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:29 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 5]}]': finished 2024-03-22T05:21:29.955 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:29 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]': finished 2024-03-22T05:21:29.955 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:29 smithi005 ceph-mon[29523]: osdmap e80: 8 total, 8 up, 8 in 2024-03-22T05:21:29.955 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:29 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:29.955 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:29 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/698140406' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6801/2160302520"}]: dispatch 2024-03-22T05:21:29.955 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:29 smithi005 ceph-mon[29523]: from='client.? ' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6801/2160302520"}]: dispatch 2024-03-22T05:21:29.955 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:29 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:29.955 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:29 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:29.955 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:29 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-22T05:21:29.955 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:29 smithi005 ceph-mon[29523]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-22T05:21:29.955 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:29 smithi005 ceph-mon[29523]: Adding iSCSI gateway http://:@172.21.15.5:5000 to Dashboard 2024-03-22T05:21:29.955 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:29 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-22T05:21:29.955 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:29 smithi005 ceph-mon[29523]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-22T05:21:29.955 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:29 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:29.955 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:29 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi005"}]: dispatch 2024-03-22T05:21:29.955 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:29 smithi005 ceph-mon[29523]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi005"}]: dispatch 2024-03-22T05:21:29.956 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:29 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:29.956 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:29 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:21:29.956 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:29 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:21:29.956 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:29 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:21:29.956 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:29 smithi005 ceph-mon[29523]: Checking dashboard <-> RGW credentials 2024-03-22T05:21:30.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:29 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/2545306374' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/2650789828"}]': finished 2024-03-22T05:21:30.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:29 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [7, 4]}]': finished 2024-03-22T05:21:30.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:29 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]': finished 2024-03-22T05:21:30.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:29 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 5]}]': finished 2024-03-22T05:21:30.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:29 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]': finished 2024-03-22T05:21:30.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:29 smithi186 ceph-mon[25720]: osdmap e80: 8 total, 8 up, 8 in 2024-03-22T05:21:30.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:29 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:30.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:29 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/698140406' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6801/2160302520"}]: dispatch 2024-03-22T05:21:30.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:29 smithi186 ceph-mon[25720]: from='client.? ' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6801/2160302520"}]: dispatch 2024-03-22T05:21:30.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:29 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:30.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:29 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:30.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:29 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-22T05:21:30.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:29 smithi186 ceph-mon[25720]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-22T05:21:30.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:29 smithi186 ceph-mon[25720]: Adding iSCSI gateway http://:@172.21.15.5:5000 to Dashboard 2024-03-22T05:21:30.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:29 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-22T05:21:30.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:29 smithi186 ceph-mon[25720]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-22T05:21:30.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:29 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:30.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:29 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi005"}]: dispatch 2024-03-22T05:21:30.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:29 smithi186 ceph-mon[25720]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi005"}]: dispatch 2024-03-22T05:21:30.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:29 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:30.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:29 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:21:30.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:29 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:21:30.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:29 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:21:30.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:29 smithi186 ceph-mon[25720]: Checking dashboard <-> RGW credentials 2024-03-22T05:21:30.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:30 smithi005 ceph-mon[24793]: from='client.? ' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6801/2160302520"}]': finished 2024-03-22T05:21:30.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:30 smithi005 ceph-mon[24793]: osdmap e81: 8 total, 8 up, 8 in 2024-03-22T05:21:30.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:30 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:30.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:30 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/3144826441' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/1122231072"}]: dispatch 2024-03-22T05:21:30.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:30 smithi005 ceph-mon[24793]: pgmap v86: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 70 MiB used, 715 GiB / 715 GiB avail; 4.7 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-22T05:21:30.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:30 smithi005 ceph-mon[29523]: from='client.? ' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6801/2160302520"}]': finished 2024-03-22T05:21:30.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:30 smithi005 ceph-mon[29523]: osdmap e81: 8 total, 8 up, 8 in 2024-03-22T05:21:30.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:30 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:30.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:30 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/3144826441' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/1122231072"}]: dispatch 2024-03-22T05:21:30.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:30 smithi005 ceph-mon[29523]: pgmap v86: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 70 MiB used, 715 GiB / 715 GiB avail; 4.7 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-22T05:21:31.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:30 smithi186 ceph-mon[25720]: from='client.? ' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6801/2160302520"}]': finished 2024-03-22T05:21:31.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:30 smithi186 ceph-mon[25720]: osdmap e81: 8 total, 8 up, 8 in 2024-03-22T05:21:31.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:30 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:21:31.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:30 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/3144826441' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/1122231072"}]: dispatch 2024-03-22T05:21:31.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:30 smithi186 ceph-mon[25720]: pgmap v86: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 70 MiB used, 715 GiB / 715 GiB avail; 4.7 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-22T05:21:32.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:31 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/3144826441' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/1122231072"}]': finished 2024-03-22T05:21:32.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:31 smithi186 ceph-mon[25720]: osdmap e82: 8 total, 8 up, 8 in 2024-03-22T05:21:32.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:31 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/1776538740' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6800/2160302520"}]: dispatch 2024-03-22T05:21:32.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:31 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/3144826441' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/1122231072"}]': finished 2024-03-22T05:21:32.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:31 smithi005 ceph-mon[24793]: osdmap e82: 8 total, 8 up, 8 in 2024-03-22T05:21:32.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:31 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/1776538740' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6800/2160302520"}]: dispatch 2024-03-22T05:21:32.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:31 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/3144826441' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/1122231072"}]': finished 2024-03-22T05:21:32.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:31 smithi005 ceph-mon[29523]: osdmap e82: 8 total, 8 up, 8 in 2024-03-22T05:21:32.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:31 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/1776538740' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6800/2160302520"}]: dispatch 2024-03-22T05:21:32.702 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:21:32 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:21:32] "GET /metrics HTTP/1.1" 200 208008 "" "Prometheus/2.33.4" 2024-03-22T05:21:33.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:32 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/1776538740' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6800/2160302520"}]': finished 2024-03-22T05:21:33.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:32 smithi186 ceph-mon[25720]: osdmap e83: 8 total, 8 up, 8 in 2024-03-22T05:21:33.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:32 smithi186 ceph-mon[25720]: pgmap v89: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 70 MiB used, 715 GiB / 715 GiB avail; 4.7 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-22T05:21:33.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:32 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/1509745738' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/2169353506"}]: dispatch 2024-03-22T05:21:33.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:32 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/1776538740' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6800/2160302520"}]': finished 2024-03-22T05:21:33.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:32 smithi005 ceph-mon[24793]: osdmap e83: 8 total, 8 up, 8 in 2024-03-22T05:21:33.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:32 smithi005 ceph-mon[24793]: pgmap v89: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 70 MiB used, 715 GiB / 715 GiB avail; 4.7 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-22T05:21:33.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:32 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/1509745738' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/2169353506"}]: dispatch 2024-03-22T05:21:33.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:32 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/1776538740' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6800/2160302520"}]': finished 2024-03-22T05:21:33.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:32 smithi005 ceph-mon[29523]: osdmap e83: 8 total, 8 up, 8 in 2024-03-22T05:21:33.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:32 smithi005 ceph-mon[29523]: pgmap v89: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 70 MiB used, 715 GiB / 715 GiB avail; 4.7 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-22T05:21:33.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:32 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/1509745738' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/2169353506"}]: dispatch 2024-03-22T05:21:33.544 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:21:33 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:21:33] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-22T05:21:33.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:33 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/1509745738' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/2169353506"}]': finished 2024-03-22T05:21:33.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:33 smithi005 ceph-mon[24793]: osdmap e84: 8 total, 8 up, 8 in 2024-03-22T05:21:33.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:33 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/3397400071' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/1346312444"}]: dispatch 2024-03-22T05:21:33.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:33 smithi005 ceph-mon[24793]: from='client.? ' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/1346312444"}]: dispatch 2024-03-22T05:21:33.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:33 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/1509745738' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/2169353506"}]': finished 2024-03-22T05:21:33.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:33 smithi005 ceph-mon[29523]: osdmap e84: 8 total, 8 up, 8 in 2024-03-22T05:21:33.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:33 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/3397400071' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/1346312444"}]: dispatch 2024-03-22T05:21:33.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:33 smithi005 ceph-mon[29523]: from='client.? ' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/1346312444"}]: dispatch 2024-03-22T05:21:33.953 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:21:33 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[49335]: level=warn ts=2024-03-22T05:21:33.521Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.186:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.186 because it doesn't contain any IP SANs" 2024-03-22T05:21:33.953 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:21:33 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[49335]: level=warn ts=2024-03-22T05:21:33.521Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.5:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.5 because it doesn't contain any IP SANs" 2024-03-22T05:21:34.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:33 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/1509745738' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/2169353506"}]': finished 2024-03-22T05:21:34.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:33 smithi186 ceph-mon[25720]: osdmap e84: 8 total, 8 up, 8 in 2024-03-22T05:21:34.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:33 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/3397400071' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/1346312444"}]: dispatch 2024-03-22T05:21:34.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:33 smithi186 ceph-mon[25720]: from='client.? ' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/1346312444"}]: dispatch 2024-03-22T05:21:35.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:34 smithi186 ceph-mon[25720]: from='client.? ' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/1346312444"}]': finished 2024-03-22T05:21:35.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:34 smithi186 ceph-mon[25720]: osdmap e85: 8 total, 8 up, 8 in 2024-03-22T05:21:35.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:34 smithi186 ceph-mon[25720]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 75 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-03-22T05:21:35.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:34 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/1510939878' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6801/294522234"}]: dispatch 2024-03-22T05:21:35.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:34 smithi005 ceph-mon[29523]: from='client.? ' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/1346312444"}]': finished 2024-03-22T05:21:35.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:34 smithi005 ceph-mon[29523]: osdmap e85: 8 total, 8 up, 8 in 2024-03-22T05:21:35.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:34 smithi005 ceph-mon[29523]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 75 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-03-22T05:21:35.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:34 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/1510939878' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6801/294522234"}]: dispatch 2024-03-22T05:21:35.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:34 smithi005 ceph-mon[24793]: from='client.? ' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/1346312444"}]': finished 2024-03-22T05:21:35.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:34 smithi005 ceph-mon[24793]: osdmap e85: 8 total, 8 up, 8 in 2024-03-22T05:21:35.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:34 smithi005 ceph-mon[24793]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 75 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-03-22T05:21:35.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:34 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/1510939878' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6801/294522234"}]: dispatch 2024-03-22T05:21:35.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:35 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/1510939878' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6801/294522234"}]': finished 2024-03-22T05:21:35.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:35 smithi005 ceph-mon[24793]: osdmap e86: 8 total, 8 up, 8 in 2024-03-22T05:21:35.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:35 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/1340966964' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/3301532190"}]: dispatch 2024-03-22T05:21:35.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:35 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/1510939878' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6801/294522234"}]': finished 2024-03-22T05:21:35.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:35 smithi005 ceph-mon[29523]: osdmap e86: 8 total, 8 up, 8 in 2024-03-22T05:21:35.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:35 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/1340966964' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/3301532190"}]: dispatch 2024-03-22T05:21:36.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:35 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/1510939878' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6801/294522234"}]': finished 2024-03-22T05:21:36.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:35 smithi186 ceph-mon[25720]: osdmap e86: 8 total, 8 up, 8 in 2024-03-22T05:21:36.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:35 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/1340966964' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/3301532190"}]: dispatch 2024-03-22T05:21:37.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:36 smithi186 ceph-mon[25720]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:21:37.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:36 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/1340966964' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/3301532190"}]': finished 2024-03-22T05:21:37.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:36 smithi186 ceph-mon[25720]: osdmap e87: 8 total, 8 up, 8 in 2024-03-22T05:21:37.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:36 smithi186 ceph-mon[25720]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 75 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-03-22T05:21:37.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:36 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/3919956493' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/2858060116"}]: dispatch 2024-03-22T05:21:37.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:36 smithi005 ceph-mon[29523]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:21:37.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:36 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/1340966964' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/3301532190"}]': finished 2024-03-22T05:21:37.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:36 smithi005 ceph-mon[29523]: osdmap e87: 8 total, 8 up, 8 in 2024-03-22T05:21:37.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:36 smithi005 ceph-mon[29523]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 75 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-03-22T05:21:37.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:36 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/3919956493' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/2858060116"}]: dispatch 2024-03-22T05:21:37.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:36 smithi005 ceph-mon[24793]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:21:37.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:36 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/1340966964' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/3301532190"}]': finished 2024-03-22T05:21:37.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:36 smithi005 ceph-mon[24793]: osdmap e87: 8 total, 8 up, 8 in 2024-03-22T05:21:37.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:36 smithi005 ceph-mon[24793]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 75 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-03-22T05:21:37.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:36 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/3919956493' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/2858060116"}]: dispatch 2024-03-22T05:21:38.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:37 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/3919956493' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/2858060116"}]': finished 2024-03-22T05:21:38.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:37 smithi186 ceph-mon[25720]: osdmap e88: 8 total, 8 up, 8 in 2024-03-22T05:21:38.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:37 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/3370600509' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6800/392664475"}]: dispatch 2024-03-22T05:21:38.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:37 smithi186 ceph-mon[25720]: from='client.? ' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6800/392664475"}]: dispatch 2024-03-22T05:21:38.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:37 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/3919956493' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/2858060116"}]': finished 2024-03-22T05:21:38.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:37 smithi005 ceph-mon[24793]: osdmap e88: 8 total, 8 up, 8 in 2024-03-22T05:21:38.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:37 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/3370600509' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6800/392664475"}]: dispatch 2024-03-22T05:21:38.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:37 smithi005 ceph-mon[24793]: from='client.? ' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6800/392664475"}]: dispatch 2024-03-22T05:21:38.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:37 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/3919956493' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/2858060116"}]': finished 2024-03-22T05:21:38.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:37 smithi005 ceph-mon[29523]: osdmap e88: 8 total, 8 up, 8 in 2024-03-22T05:21:38.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:37 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/3370600509' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6800/392664475"}]: dispatch 2024-03-22T05:21:38.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:37 smithi005 ceph-mon[29523]: from='client.? ' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6800/392664475"}]: dispatch 2024-03-22T05:21:38.796 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:38 smithi005 ceph-mon[24793]: from='client.? ' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6800/392664475"}]': finished 2024-03-22T05:21:38.796 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:38 smithi005 ceph-mon[24793]: osdmap e89: 8 total, 8 up, 8 in 2024-03-22T05:21:38.796 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:38 smithi005 ceph-mon[24793]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 75 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:21:38.797 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:38 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/3135483762' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6801/392664475"}]: dispatch 2024-03-22T05:21:38.797 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:38 smithi005 ceph-mon[29523]: from='client.? ' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6800/392664475"}]': finished 2024-03-22T05:21:38.797 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:38 smithi005 ceph-mon[29523]: osdmap e89: 8 total, 8 up, 8 in 2024-03-22T05:21:38.797 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:38 smithi005 ceph-mon[29523]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 75 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:21:38.797 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:38 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/3135483762' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6801/392664475"}]: dispatch 2024-03-22T05:21:39.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:38 smithi186 ceph-mon[25720]: from='client.? ' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6800/392664475"}]': finished 2024-03-22T05:21:39.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:38 smithi186 ceph-mon[25720]: osdmap e89: 8 total, 8 up, 8 in 2024-03-22T05:21:39.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:38 smithi186 ceph-mon[25720]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 75 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:21:39.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:38 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/3135483762' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6801/392664475"}]: dispatch 2024-03-22T05:21:40.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:39 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/3135483762' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6801/392664475"}]': finished 2024-03-22T05:21:40.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:39 smithi186 ceph-mon[25720]: osdmap e90: 8 total, 8 up, 8 in 2024-03-22T05:21:40.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:39 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/3257891970' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/1891765076"}]: dispatch 2024-03-22T05:21:40.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:39 smithi186 ceph-mon[25720]: from='client.? ' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/1891765076"}]: dispatch 2024-03-22T05:21:40.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:39 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/3135483762' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6801/392664475"}]': finished 2024-03-22T05:21:40.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:39 smithi005 ceph-mon[24793]: osdmap e90: 8 total, 8 up, 8 in 2024-03-22T05:21:40.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:39 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/3257891970' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/1891765076"}]: dispatch 2024-03-22T05:21:40.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:39 smithi005 ceph-mon[24793]: from='client.? ' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/1891765076"}]: dispatch 2024-03-22T05:21:40.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:39 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/3135483762' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6801/392664475"}]': finished 2024-03-22T05:21:40.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:39 smithi005 ceph-mon[29523]: osdmap e90: 8 total, 8 up, 8 in 2024-03-22T05:21:40.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:39 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/3257891970' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/1891765076"}]: dispatch 2024-03-22T05:21:40.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:39 smithi005 ceph-mon[29523]: from='client.? ' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/1891765076"}]: dispatch 2024-03-22T05:21:41.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:40 smithi005 ceph-mon[24793]: from='client.? ' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/1891765076"}]': finished 2024-03-22T05:21:41.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:40 smithi005 ceph-mon[24793]: osdmap e91: 8 total, 8 up, 8 in 2024-03-22T05:21:41.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:40 smithi005 ceph-mon[24793]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 75 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:21:41.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:40 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/1484911972' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/2331329527"}]: dispatch 2024-03-22T05:21:41.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:40 smithi005 ceph-mon[29523]: from='client.? ' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/1891765076"}]': finished 2024-03-22T05:21:41.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:40 smithi005 ceph-mon[29523]: osdmap e91: 8 total, 8 up, 8 in 2024-03-22T05:21:41.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:40 smithi005 ceph-mon[29523]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 75 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:21:41.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:40 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/1484911972' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/2331329527"}]: dispatch 2024-03-22T05:21:41.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:40 smithi186 ceph-mon[25720]: from='client.? ' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/1891765076"}]': finished 2024-03-22T05:21:41.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:40 smithi186 ceph-mon[25720]: osdmap e91: 8 total, 8 up, 8 in 2024-03-22T05:21:41.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:40 smithi186 ceph-mon[25720]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 75 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:21:41.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:40 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/1484911972' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/2331329527"}]: dispatch 2024-03-22T05:21:42.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:41 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/1484911972' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/2331329527"}]': finished 2024-03-22T05:21:42.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:41 smithi005 ceph-mon[24793]: osdmap e92: 8 total, 8 up, 8 in 2024-03-22T05:21:42.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:41 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/777315997' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/789815545"}]: dispatch 2024-03-22T05:21:42.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:41 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/1484911972' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/2331329527"}]': finished 2024-03-22T05:21:42.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:41 smithi005 ceph-mon[29523]: osdmap e92: 8 total, 8 up, 8 in 2024-03-22T05:21:42.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:41 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/777315997' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/789815545"}]: dispatch 2024-03-22T05:21:42.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:41 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/1484911972' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/2331329527"}]': finished 2024-03-22T05:21:42.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:41 smithi186 ceph-mon[25720]: osdmap e92: 8 total, 8 up, 8 in 2024-03-22T05:21:42.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:41 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/777315997' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/789815545"}]: dispatch 2024-03-22T05:21:42.702 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:21:42 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:21:42] "GET /metrics HTTP/1.1" 200 208008 "" "Prometheus/2.33.4" 2024-03-22T05:21:43.192 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:42 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/777315997' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/789815545"}]': finished 2024-03-22T05:21:43.192 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:42 smithi186 ceph-mon[25720]: osdmap e93: 8 total, 8 up, 8 in 2024-03-22T05:21:43.192 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:42 smithi186 ceph-mon[25720]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 75 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:21:43.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:42 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/777315997' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/789815545"}]': finished 2024-03-22T05:21:43.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:42 smithi005 ceph-mon[24793]: osdmap e93: 8 total, 8 up, 8 in 2024-03-22T05:21:43.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:42 smithi005 ceph-mon[24793]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 75 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:21:43.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:42 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/777315997' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/789815545"}]': finished 2024-03-22T05:21:43.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:42 smithi005 ceph-mon[29523]: osdmap e93: 8 total, 8 up, 8 in 2024-03-22T05:21:43.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:42 smithi005 ceph-mon[29523]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 75 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:21:43.543 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:21:43 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:21:43] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-22T05:21:43.952 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:21:43 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[49335]: level=error ts=2024-03-22T05:21:43.507Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.5:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.5 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.186:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.186 because it doesn't contain any IP SANs" 2024-03-22T05:21:44.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:44 smithi005 ceph-mon[24793]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 1 op/s 2024-03-22T05:21:44.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:44 smithi005 ceph-mon[29523]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 1 op/s 2024-03-22T05:21:44.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:44 smithi186 ceph-mon[25720]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 1 op/s 2024-03-22T05:21:47.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:47 smithi186 ceph-mon[25720]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:21:47.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:47 smithi186 ceph-mon[25720]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 824 B/s rd, 0 op/s 2024-03-22T05:21:47.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:47 smithi005 ceph-mon[29523]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:21:47.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:47 smithi005 ceph-mon[29523]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 824 B/s rd, 0 op/s 2024-03-22T05:21:47.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:47 smithi005 ceph-mon[24793]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:21:47.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:47 smithi005 ceph-mon[24793]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 824 B/s rd, 0 op/s 2024-03-22T05:21:49.348 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:49 smithi186 ceph-mon[25720]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-22T05:21:49.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:49 smithi005 ceph-mon[24793]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-22T05:21:49.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:49 smithi005 ceph-mon[29523]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-22T05:21:51.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:51 smithi005 ceph-mon[24793]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:21:51.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:51 smithi005 ceph-mon[29523]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:21:51.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:51 smithi186 ceph-mon[25720]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:21:52.702 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:21:52 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:21:52] "GET /metrics HTTP/1.1" 200 208016 "" "Prometheus/2.33.4" 2024-03-22T05:21:53.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:53 smithi005 ceph-mon[29523]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1004 B/s rd, 0 op/s 2024-03-22T05:21:53.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:53 smithi005 ceph-mon[24793]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1004 B/s rd, 0 op/s 2024-03-22T05:21:53.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:53 smithi186 ceph-mon[25720]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1004 B/s rd, 0 op/s 2024-03-22T05:21:53.544 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:21:53 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:21:53] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-22T05:21:54.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:54 smithi005 ceph-mon[24793]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:21:54.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:54 smithi005 ceph-mon[29523]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:21:54.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:54 smithi186 ceph-mon[25720]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:21:57.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:57 smithi005 ceph-mon[24793]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:21:57.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:57 smithi005 ceph-mon[24793]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:21:57.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:57 smithi005 ceph-mon[29523]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:21:57.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:57 smithi005 ceph-mon[29523]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:21:57.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:57 smithi186 ceph-mon[25720]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:21:57.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:57 smithi186 ceph-mon[25720]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:21:59.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:21:59 smithi005 ceph-mon[24793]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:21:59.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:21:59 smithi005 ceph-mon[29523]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:21:59.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:21:59 smithi186 ceph-mon[25720]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:22:01.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:22:01 smithi005 ceph-mon[24793]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:22:01.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:22:01 smithi005 ceph-mon[29523]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:22:01.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:22:01 smithi186 ceph-mon[25720]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:22:02.702 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:22:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:22:02] "GET /metrics HTTP/1.1" 200 207983 "" "Prometheus/2.33.4" 2024-03-22T05:22:03.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:22:03 smithi005 ceph-mon[24793]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:22:03.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:22:03 smithi005 ceph-mon[29523]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:22:03.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:22:03 smithi186 ceph-mon[25720]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:22:03.544 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:22:03 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:22:03] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-22T05:22:04.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:22:04 smithi005 ceph-mon[24793]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:22:04.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:22:04 smithi005 ceph-mon[29523]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:22:04.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:22:04 smithi186 ceph-mon[25720]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:22:07.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:22:07 smithi005 ceph-mon[29523]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:22:07.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:22:07 smithi005 ceph-mon[29523]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:22:07.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:22:07 smithi005 ceph-mon[24793]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:22:07.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:22:07 smithi005 ceph-mon[24793]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:22:07.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:22:07 smithi186 ceph-mon[25720]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:22:07.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:22:07 smithi186 ceph-mon[25720]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:22:09.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:22:09 smithi005 ceph-mon[29523]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:22:09.455 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:22:09 smithi005 ceph-mon[24793]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:22:09.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:22:09 smithi186 ceph-mon[25720]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:22:11.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:22:11 smithi005 ceph-mon[29523]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:22:11.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:22:11 smithi005 ceph-mon[24793]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:22:11.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:22:11 smithi186 ceph-mon[25720]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:22:12.702 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:22:12 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:22:12] "GET /metrics HTTP/1.1" 200 207983 "" "Prometheus/2.33.4" 2024-03-22T05:22:13.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:22:13 smithi005 ceph-mon[29523]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:22:13.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:22:13 smithi005 ceph-mon[24793]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:22:13.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:22:13 smithi186 ceph-mon[25720]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:22:13.544 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:22:13 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:22:13] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-22T05:22:14.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:22:14 smithi005 ceph-mon[24793]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:22:14.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:22:14 smithi005 ceph-mon[29523]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:22:14.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:22:14 smithi186 ceph-mon[25720]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:22:17.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:22:17 smithi005 ceph-mon[24793]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:22:17.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:22:17 smithi005 ceph-mon[24793]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:22:17.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:22:17 smithi005 ceph-mon[29523]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:22:17.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:22:17 smithi005 ceph-mon[29523]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:22:17.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:22:17 smithi186 ceph-mon[25720]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:22:17.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:22:17 smithi186 ceph-mon[25720]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:22:19.345 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:22:19 smithi186 ceph-mon[25720]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:22:19.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:22:19 smithi005 ceph-mon[24793]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:22:19.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:22:19 smithi005 ceph-mon[29523]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:22:21.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:22:21 smithi005 ceph-mon[24793]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:22:21.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:22:21 smithi005 ceph-mon[29523]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:22:21.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:22:21 smithi186 ceph-mon[25720]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:22:22.702 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:22:22 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:22:22] "GET /metrics HTTP/1.1" 200 207937 "" "Prometheus/2.33.4" 2024-03-22T05:22:23.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:22:23 smithi005 ceph-mon[24793]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:22:23.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:22:23 smithi005 ceph-mon[29523]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:22:23.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:22:23 smithi186 ceph-mon[25720]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:22:23.544 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:22:23 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:22:23] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-22T05:22:24.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:22:24 smithi005 ceph-mon[24793]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:22:24.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:22:24 smithi005 ceph-mon[29523]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:22:24.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:22:24 smithi186 ceph-mon[25720]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:22:27.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:22:27 smithi005 ceph-mon[29523]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:22:27.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:22:27 smithi005 ceph-mon[29523]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:22:27.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:22:27 smithi005 ceph-mon[24793]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:22:27.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:22:27 smithi005 ceph-mon[24793]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:22:27.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:22:27 smithi186 ceph-mon[25720]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:22:27.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:22:27 smithi186 ceph-mon[25720]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:22:28.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:22:28 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-22T05:22:28.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:22:28 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-22T05:22:28.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:22:28 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-22T05:22:28.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:22:28 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-22T05:22:28.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:22:28 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-22T05:22:28.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:22:28 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-22T05:22:29.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:22:29 smithi005 ceph-mon[24793]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:22:29.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:22:29 smithi005 ceph-mon[29523]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:22:29.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:22:29 smithi186 ceph-mon[25720]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:22:30.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:22:30 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:22:30.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:22:30 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:22:30.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:22:30 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:22:30.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:22:30 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:22:30.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:22:30 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:22:30.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:22:30 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:22:30.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:22:30 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:22:30.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:22:30 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:22:30.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:22:30 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:22:31.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:22:31 smithi005 ceph-mon[24793]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:22:31.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:22:31 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:22:31.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:22:31 smithi005 ceph-mon[29523]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:22:31.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:22:31 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:22:31.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:22:31 smithi186 ceph-mon[25720]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:22:31.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:22:31 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:22:32.702 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:22:32 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:22:32] "GET /metrics HTTP/1.1" 200 207926 "" "Prometheus/2.33.4" 2024-03-22T05:22:33.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:22:33 smithi005 ceph-mon[24793]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:22:33.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:22:33 smithi005 ceph-mon[29523]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:22:33.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:22:33 smithi186 ceph-mon[25720]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:22:33.544 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:22:33 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:22:33] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-22T05:22:34.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:22:34 smithi005 ceph-mon[24793]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:22:34.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:22:34 smithi005 ceph-mon[29523]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:22:34.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:22:34 smithi186 ceph-mon[25720]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:22:37.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:22:37 smithi005 ceph-mon[24793]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:22:37.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:22:37 smithi005 ceph-mon[24793]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:22:37.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:22:37 smithi005 ceph-mon[29523]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:22:37.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:22:37 smithi005 ceph-mon[29523]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:22:37.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:22:37 smithi186 ceph-mon[25720]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:22:37.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:22:37 smithi186 ceph-mon[25720]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:22:39.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:22:39 smithi005 ceph-mon[29523]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:22:39.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:22:39 smithi005 ceph-mon[24793]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:22:39.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:22:39 smithi186 ceph-mon[25720]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:22:41.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:22:41 smithi005 ceph-mon[24793]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:22:41.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:22:41 smithi005 ceph-mon[29523]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:22:41.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:22:41 smithi186 ceph-mon[25720]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:22:42.702 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:22:42 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:22:42] "GET /metrics HTTP/1.1" 200 207926 "" "Prometheus/2.33.4" 2024-03-22T05:22:43.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:22:43 smithi005 ceph-mon[29523]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:22:43.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:22:43 smithi005 ceph-mon[24793]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:22:43.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:22:43 smithi186 ceph-mon[25720]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:22:43.544 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:22:43 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:22:43] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-22T05:22:44.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:22:44 smithi005 ceph-mon[29523]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:22:44.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:22:44 smithi005 ceph-mon[24793]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:22:44.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:22:44 smithi186 ceph-mon[25720]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:22:47.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:22:47 smithi005 ceph-mon[29523]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:22:47.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:22:47 smithi005 ceph-mon[29523]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:22:47.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:22:47 smithi005 ceph-mon[24793]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:22:47.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:22:47 smithi005 ceph-mon[24793]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:22:47.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:22:47 smithi186 ceph-mon[25720]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:22:47.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:22:47 smithi186 ceph-mon[25720]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:22:49.345 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:22:49 smithi186 ceph-mon[25720]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:22:49.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:22:49 smithi005 ceph-mon[29523]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:22:49.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:22:49 smithi005 ceph-mon[24793]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:22:51.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:22:51 smithi005 ceph-mon[29523]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:22:51.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:22:51 smithi005 ceph-mon[24793]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:22:51.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:22:51 smithi186 ceph-mon[25720]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:22:52.702 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:22:52 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:22:52] "GET /metrics HTTP/1.1" 200 207925 "" "Prometheus/2.33.4" 2024-03-22T05:22:53.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:22:53 smithi005 ceph-mon[24793]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:22:53.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:22:53 smithi005 ceph-mon[29523]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:22:53.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:22:53 smithi186 ceph-mon[25720]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:22:53.544 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:22:53 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:22:53] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-22T05:22:54.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:22:54 smithi005 ceph-mon[24793]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:22:54.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:22:54 smithi005 ceph-mon[29523]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:22:54.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:22:54 smithi186 ceph-mon[25720]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:22:57.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:22:57 smithi005 ceph-mon[29523]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:22:57.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:22:57 smithi005 ceph-mon[29523]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:22:57.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:22:57 smithi005 ceph-mon[24793]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:22:57.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:22:57 smithi005 ceph-mon[24793]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:22:57.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:22:57 smithi186 ceph-mon[25720]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:22:57.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:22:57 smithi186 ceph-mon[25720]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:22:59.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:22:59 smithi005 ceph-mon[24793]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:22:59.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:22:59 smithi005 ceph-mon[29523]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:22:59.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:22:59 smithi186 ceph-mon[25720]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:23:01.455 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:23:01 smithi005 ceph-mon[29523]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:23:01.455 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:23:01 smithi005 ceph-mon[24793]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:23:01.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:23:01 smithi186 ceph-mon[25720]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:23:02.705 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:23:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:23:02] "GET /metrics HTTP/1.1" 200 207935 "" "Prometheus/2.33.4" 2024-03-22T05:23:03.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:23:03 smithi005 ceph-mon[24793]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:23:03.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:23:03 smithi005 ceph-mon[29523]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:23:03.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:23:03 smithi186 ceph-mon[25720]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:23:03.544 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:23:03 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:23:03] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-22T05:23:04.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:23:04 smithi005 ceph-mon[24793]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:23:04.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:23:04 smithi005 ceph-mon[29523]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:23:04.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:23:04 smithi186 ceph-mon[25720]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:23:07.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:23:07 smithi005 ceph-mon[24793]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:23:07.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:23:07 smithi005 ceph-mon[24793]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:23:07.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:23:07 smithi005 ceph-mon[29523]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:23:07.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:23:07 smithi005 ceph-mon[29523]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:23:07.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:23:07 smithi186 ceph-mon[25720]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:23:07.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:23:07 smithi186 ceph-mon[25720]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:23:09.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:23:09 smithi005 ceph-mon[24793]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:23:09.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:23:09 smithi005 ceph-mon[29523]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:23:09.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:23:09 smithi186 ceph-mon[25720]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:23:11.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:23:11 smithi005 ceph-mon[24793]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:23:11.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:23:11 smithi005 ceph-mon[29523]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:23:11.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:23:11 smithi186 ceph-mon[25720]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:23:12.702 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:23:12 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:23:12] "GET /metrics HTTP/1.1" 200 207935 "" "Prometheus/2.33.4" 2024-03-22T05:23:13.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:23:13 smithi005 ceph-mon[29523]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:23:13.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:23:13 smithi005 ceph-mon[24793]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:23:13.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:23:13 smithi186 ceph-mon[25720]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:23:13.544 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:23:13 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:23:13] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-22T05:23:14.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:23:14 smithi005 ceph-mon[29523]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:23:14.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:23:14 smithi005 ceph-mon[24793]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:23:14.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:23:14 smithi186 ceph-mon[25720]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:23:17.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:23:17 smithi005 ceph-mon[29523]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:23:17.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:23:17 smithi005 ceph-mon[29523]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:23:17.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:23:17 smithi005 ceph-mon[24793]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:23:17.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:23:17 smithi005 ceph-mon[24793]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:23:17.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:23:17 smithi186 ceph-mon[25720]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:23:17.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:23:17 smithi186 ceph-mon[25720]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:23:19.346 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:23:19 smithi186 ceph-mon[25720]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:23:19.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:23:19 smithi005 ceph-mon[29523]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:23:19.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:23:19 smithi005 ceph-mon[24793]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:23:21.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:23:21 smithi005 ceph-mon[29523]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:23:21.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:23:21 smithi005 ceph-mon[24793]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:23:21.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:23:21 smithi186 ceph-mon[25720]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:23:22.702 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:23:22 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:23:22] "GET /metrics HTTP/1.1" 200 207941 "" "Prometheus/2.33.4" 2024-03-22T05:23:23.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:23:23 smithi005 ceph-mon[24793]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:23:23.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:23:23 smithi005 ceph-mon[29523]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:23:23.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:23:23 smithi186 ceph-mon[25720]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:23:23.544 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:23:23 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:23:23] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-22T05:23:24.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:23:24 smithi005 ceph-mon[24793]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:23:24.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:23:24 smithi005 ceph-mon[29523]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:23:24.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:23:24 smithi186 ceph-mon[25720]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:23:27.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:23:27 smithi005 ceph-mon[24793]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:23:27.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:23:27 smithi005 ceph-mon[24793]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:23:27.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:23:27 smithi005 ceph-mon[29523]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:23:27.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:23:27 smithi005 ceph-mon[29523]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:23:27.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:23:27 smithi186 ceph-mon[25720]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:23:27.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:23:27 smithi186 ceph-mon[25720]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:23:28.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:23:28 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-22T05:23:28.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:23:28 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-22T05:23:28.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:23:28 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-22T05:23:28.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:23:28 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-22T05:23:28.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:23:28 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-22T05:23:28.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:23:28 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-22T05:23:29.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:23:29 smithi005 ceph-mon[29523]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:23:29.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:23:29 smithi005 ceph-mon[24793]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:23:29.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:23:29 smithi186 ceph-mon[25720]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:23:31.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:23:31 smithi005 ceph-mon[24793]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:23:31.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:23:31 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:23:31.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:23:31 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:23:31.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:23:31 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:23:31.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:23:31 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:23:31.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:23:31 smithi005 ceph-mon[29523]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:23:31.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:23:31 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:23:31.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:23:31 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:23:31.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:23:31 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:23:31.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:23:31 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:23:31.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:23:31 smithi186 ceph-mon[25720]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:23:31.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:23:31 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:23:31.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:23:31 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:23:31.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:23:31 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:23:31.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:23:31 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:23:32.702 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:23:32 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:23:32] "GET /metrics HTTP/1.1" 200 207932 "" "Prometheus/2.33.4" 2024-03-22T05:23:33.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:23:33 smithi005 ceph-mon[24793]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:23:33.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:23:33 smithi005 ceph-mon[29523]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:23:33.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:23:33 smithi186 ceph-mon[25720]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:23:33.544 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:23:33 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:23:33] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-22T05:23:34.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:23:34 smithi005 ceph-mon[24793]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:23:34.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:23:34 smithi005 ceph-mon[29523]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:23:34.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:23:34 smithi186 ceph-mon[25720]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:23:37.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:23:37 smithi005 ceph-mon[24793]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:23:37.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:23:37 smithi005 ceph-mon[24793]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:23:37.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:23:37 smithi005 ceph-mon[29523]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:23:37.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:23:37 smithi005 ceph-mon[29523]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:23:37.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:23:37 smithi186 ceph-mon[25720]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:23:37.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:23:37 smithi186 ceph-mon[25720]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:23:39.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:23:39 smithi005 ceph-mon[29523]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:23:39.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:23:39 smithi005 ceph-mon[24793]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:23:39.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:23:39 smithi186 ceph-mon[25720]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:23:41.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:23:41 smithi005 ceph-mon[29523]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:23:41.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:23:41 smithi005 ceph-mon[24793]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:23:41.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:23:41 smithi186 ceph-mon[25720]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:23:42.702 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:23:42 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:23:42] "GET /metrics HTTP/1.1" 200 207932 "" "Prometheus/2.33.4" 2024-03-22T05:23:43.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:23:43 smithi005 ceph-mon[24793]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:23:43.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:23:43 smithi005 ceph-mon[29523]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:23:43.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:23:43 smithi186 ceph-mon[25720]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:23:43.544 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:23:43 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:23:43] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-22T05:23:44.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:23:44 smithi005 ceph-mon[24793]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:23:44.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:23:44 smithi005 ceph-mon[29523]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:23:44.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:23:44 smithi186 ceph-mon[25720]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:23:47.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:23:47 smithi005 ceph-mon[29523]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:23:47.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:23:47 smithi005 ceph-mon[29523]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:23:47.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:23:47 smithi005 ceph-mon[24793]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:23:47.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:23:47 smithi005 ceph-mon[24793]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:23:47.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:23:47 smithi186 ceph-mon[25720]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:23:47.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:23:47 smithi186 ceph-mon[25720]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:23:49.346 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:23:49 smithi186 ceph-mon[25720]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:23:49.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:23:49 smithi005 ceph-mon[24793]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:23:49.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:23:49 smithi005 ceph-mon[29523]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:23:51.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:23:51 smithi005 ceph-mon[29523]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:23:51.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:23:51 smithi005 ceph-mon[24793]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:23:51.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:23:51 smithi186 ceph-mon[25720]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:23:52.702 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:23:52 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:23:52] "GET /metrics HTTP/1.1" 200 207927 "" "Prometheus/2.33.4" 2024-03-22T05:23:53.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:23:53 smithi005 ceph-mon[29523]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:23:53.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:23:53 smithi005 ceph-mon[24793]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:23:53.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:23:53 smithi186 ceph-mon[25720]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:23:53.545 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:23:53 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:23:53] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-22T05:23:54.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:23:54 smithi005 ceph-mon[29523]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:23:54.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:23:54 smithi005 ceph-mon[24793]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:23:54.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:23:54 smithi186 ceph-mon[25720]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:23:57.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:23:57 smithi005 ceph-mon[24793]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:23:57.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:23:57 smithi005 ceph-mon[24793]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:23:57.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:23:57 smithi005 ceph-mon[29523]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:23:57.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:23:57 smithi005 ceph-mon[29523]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:23:57.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:23:57 smithi186 ceph-mon[25720]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:23:57.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:23:57 smithi186 ceph-mon[25720]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:23:59.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:23:59 smithi005 ceph-mon[24793]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:23:59.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:23:59 smithi005 ceph-mon[29523]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:23:59.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:23:59 smithi186 ceph-mon[25720]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:24:01.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:24:01 smithi005 ceph-mon[24793]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:24:01.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:24:01 smithi005 ceph-mon[29523]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:24:01.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:24:01 smithi186 ceph-mon[25720]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:24:02.702 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:24:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:24:02] "GET /metrics HTTP/1.1" 200 207874 "" "Prometheus/2.33.4" 2024-03-22T05:24:03.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:24:03 smithi005 ceph-mon[29523]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:24:03.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:24:03 smithi005 ceph-mon[24793]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:24:03.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:24:03 smithi186 ceph-mon[25720]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:24:03.544 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:24:03 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:24:03] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-22T05:24:04.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:24:04 smithi005 ceph-mon[24793]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:24:04.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:24:04 smithi005 ceph-mon[29523]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:24:04.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:24:04 smithi186 ceph-mon[25720]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:24:07.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:24:07 smithi005 ceph-mon[29523]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:24:07.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:24:07 smithi005 ceph-mon[29523]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:24:07.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:24:07 smithi005 ceph-mon[24793]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:24:07.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:24:07 smithi005 ceph-mon[24793]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:24:07.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:24:07 smithi186 ceph-mon[25720]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:24:07.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:24:07 smithi186 ceph-mon[25720]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:24:09.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:24:09 smithi005 ceph-mon[29523]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:24:09.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:24:09 smithi005 ceph-mon[24793]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:24:09.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:24:09 smithi186 ceph-mon[25720]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:24:11.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:24:11 smithi005 ceph-mon[29523]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:24:11.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:24:11 smithi005 ceph-mon[24793]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:24:11.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:24:11 smithi186 ceph-mon[25720]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:24:12.702 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:24:12 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:24:12] "GET /metrics HTTP/1.1" 200 207874 "" "Prometheus/2.33.4" 2024-03-22T05:24:13.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:24:13 smithi005 ceph-mon[24793]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:24:13.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:24:13 smithi005 ceph-mon[29523]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:24:13.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:24:13 smithi186 ceph-mon[25720]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:24:13.544 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:24:13 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:24:13] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-22T05:24:14.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:24:14 smithi005 ceph-mon[29523]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:24:14.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:24:14 smithi005 ceph-mon[24793]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:24:14.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:24:14 smithi186 ceph-mon[25720]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:24:17.367 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:24:17 smithi005 ceph-mon[24793]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:24:17.367 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:24:17 smithi005 ceph-mon[24793]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:24:17.367 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:24:17 smithi005 ceph-mon[29523]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:24:17.367 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:24:17 smithi005 ceph-mon[29523]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:24:17.396 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-03-22T05:24:17.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:24:17 smithi186 ceph-mon[25720]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:24:17.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:24:17 smithi186 ceph-mon[25720]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:24:19.252 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:24:19 smithi005 ceph-mon[24793]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:24:19.252 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:24:19 smithi005 ceph-mon[29523]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:24:19.346 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:24:19 smithi186 ceph-mon[25720]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:24:19.843 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-03-22T05:24:21.345 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:24:21 smithi005 ceph-mon[24793]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:24:21.346 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:24:21 smithi005 ceph-mon[29523]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:24:21.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:24:21 smithi186 ceph-mon[25720]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:24:22.361 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph config set global log_to_journald false --force' 2024-03-22T05:24:22.702 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:24:22 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:24:22] "GET /metrics HTTP/1.1" 200 207876 "" "Prometheus/2.33.4" 2024-03-22T05:24:23.352 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:24:23 smithi005 ceph-mon[24793]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:24:23.353 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:24:23 smithi005 ceph-mon[29523]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:24:23.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:24:23 smithi186 ceph-mon[25720]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:24:23.544 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:24:23 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:24:23] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-22T05:24:24.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:24:24 smithi005 ceph-mon[24793]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:24:24.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:24:24 smithi005 ceph-mon[29523]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:24:24.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:24:24 smithi186 ceph-mon[25720]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:24:25.019 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph orch ps' 2024-03-22T05:24:26.861 INFO:teuthology.orchestra.run.smithi005.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T05:24:26.861 INFO:teuthology.orchestra.run.smithi005.stdout:alertmanager.a smithi005 *:9093,9094 running (3m) 2m ago 3m 16.7M - ba2b418f427c ee43e9b1c427 2024-03-22T05:24:26.861 INFO:teuthology.orchestra.run.smithi005.stdout:grafana.a smithi186 *:3000 running (3m) 2m ago 3m 49.6M - 8.3.5 dad864ee21e9 27324e98a8bb 2024-03-22T05:24:26.861 INFO:teuthology.orchestra.run.smithi005.stdout:iscsi.foo.smithi005.mcdwbp smithi005 running (3m) 2m ago 3m 63.6M - 3.5 e1d6a67b021e a2293f952ef5 2024-03-22T05:24:26.861 INFO:teuthology.orchestra.run.smithi005.stdout:mgr.x smithi186 *:8443 running (8m) 2m ago 8m 411M - 17.2.0 e1d6a67b021e febfc27d3c00 2024-03-22T05:24:26.861 INFO:teuthology.orchestra.run.smithi005.stdout:mgr.y smithi005 *:9283 running (10m) 2m ago 10m 454M - 17.2.0 e1d6a67b021e fcee4f81e3ee 2024-03-22T05:24:26.862 INFO:teuthology.orchestra.run.smithi005.stdout:mon.a smithi005 running (10m) 2m ago 10m 47.1M 2048M 17.2.0 e1d6a67b021e 38e0bba4bb03 2024-03-22T05:24:26.862 INFO:teuthology.orchestra.run.smithi005.stdout:mon.b smithi186 running (8m) 2m ago 8m 38.6M 2048M 17.2.0 e1d6a67b021e 323f53f932f6 2024-03-22T05:24:26.862 INFO:teuthology.orchestra.run.smithi005.stdout:mon.c smithi005 running (8m) 2m ago 8m 37.7M 2048M 17.2.0 e1d6a67b021e 00ec24d38612 2024-03-22T05:24:26.862 INFO:teuthology.orchestra.run.smithi005.stdout:node-exporter.a smithi005 *:9100 running (4m) 2m ago 4m 13.3M - 1dbe0e931976 1ee16c40186b 2024-03-22T05:24:26.862 INFO:teuthology.orchestra.run.smithi005.stdout:node-exporter.b smithi186 *:9100 running (4m) 2m ago 4m 17.4M - 1dbe0e931976 e1c552dad930 2024-03-22T05:24:26.862 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0 smithi005 running (8m) 2m ago 8m 47.7M 4017M 17.2.0 e1d6a67b021e ef5c7053dd37 2024-03-22T05:24:26.862 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1 smithi005 running (7m) 2m ago 7m 49.6M 4017M 17.2.0 e1d6a67b021e a1e7ee1017b6 2024-03-22T05:24:26.862 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2 smithi005 running (7m) 2m ago 7m 42.1M 4017M 17.2.0 e1d6a67b021e 462ea63ad0e5 2024-03-22T05:24:26.862 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3 smithi005 running (6m) 2m ago 6m 48.4M 4017M 17.2.0 e1d6a67b021e b25432a7239b 2024-03-22T05:24:26.862 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4 smithi186 running (6m) 2m ago 6m 45.9M 4272M 17.2.0 e1d6a67b021e cd963671f340 2024-03-22T05:24:26.862 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5 smithi186 running (6m) 2m ago 6m 43.7M 4272M 17.2.0 e1d6a67b021e f98c87f1a2cd 2024-03-22T05:24:26.862 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6 smithi186 running (5m) 2m ago 5m 43.5M 4272M 17.2.0 e1d6a67b021e 704ac49c68c4 2024-03-22T05:24:26.862 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7 smithi186 running (5m) 2m ago 5m 45.8M 4272M 17.2.0 e1d6a67b021e f415b90edf63 2024-03-22T05:24:26.862 INFO:teuthology.orchestra.run.smithi005.stdout:prometheus.a smithi186 *:9095 running (3m) 2m ago 4m 42.6M - 514e6a882f6e f520626945ac 2024-03-22T05:24:26.863 INFO:teuthology.orchestra.run.smithi005.stdout:rgw.foo.smithi005.exdhqu smithi005 *:8000 running (3m) 2m ago 3m 72.2M - 17.2.0 e1d6a67b021e 6bb27fe11553 2024-03-22T05:24:26.863 INFO:teuthology.orchestra.run.smithi005.stdout:rgw.foo.smithi186.mcgodk smithi186 *:8000 running (3m) 2m ago 3m 69.7M - 17.2.0 e1d6a67b021e 19b5c20a34f7 2024-03-22T05:24:27.180 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:24:27 smithi005 ceph-mon[24793]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:24:27.180 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:24:27 smithi005 ceph-mon[24793]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:24:27.180 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:24:27 smithi005 ceph-mon[29523]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:24:27.180 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:24:27 smithi005 ceph-mon[29523]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:24:27.445 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph versions' 2024-03-22T05:24:27.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:24:27 smithi186 ceph-mon[25720]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:24:27.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:24:27 smithi186 ceph-mon[25720]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:24:28.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:24:28 smithi005 ceph-mon[24793]: from='client.24868 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:24:28.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:24:28 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-22T05:24:28.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:24:28 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-22T05:24:28.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:24:28 smithi005 ceph-mon[29523]: from='client.24868 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:24:28.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:24:28 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-22T05:24:28.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:24:28 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-22T05:24:28.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:24:28 smithi186 ceph-mon[25720]: from='client.24868 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:24:28.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:24:28 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-22T05:24:28.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:24:28 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-22T05:24:29.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:24:29 smithi005 ceph-mon[24793]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:24:29.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:24:29 smithi005 ceph-mon[29523]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:24:29.457 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-03-22T05:24:29.457 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": { 2024-03-22T05:24:29.458 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-03-22T05:24:29.458 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:24:29.458 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr": { 2024-03-22T05:24:29.458 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-22T05:24:29.458 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:24:29.458 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": { 2024-03-22T05:24:29.458 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-22T05:24:29.458 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:24:29.458 INFO:teuthology.orchestra.run.smithi005.stdout: "mds": {}, 2024-03-22T05:24:29.458 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw": { 2024-03-22T05:24:29.458 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-22T05:24:29.458 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:24:29.459 INFO:teuthology.orchestra.run.smithi005.stdout: "overall": { 2024-03-22T05:24:29.459 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 15 2024-03-22T05:24:29.459 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-03-22T05:24:29.459 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-03-22T05:24:29.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:24:29 smithi186 ceph-mon[25720]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:24:30.076 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph -s' 2024-03-22T05:24:30.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:24:30 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/386939798' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:24:30.388 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:24:30 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/386939798' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:24:30.388 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:24:30 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/386939798' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:24:31.335 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:24:31 smithi005 ceph-mon[24793]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:24:31.335 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:24:31 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:24:31.335 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:24:31 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:24:31.335 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:24:31 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:24:31.335 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:24:31 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:24:31.335 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:24:31 smithi005 ceph-mon[29523]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:24:31.335 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:24:31 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:24:31.335 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:24:31 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:24:31.335 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:24:31 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:24:31.336 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:24:31 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:24:31.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:24:31 smithi186 ceph-mon[25720]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:24:31.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:24:31 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:24:31.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:24:31 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:24:31.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:24:31 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:24:31.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:24:31 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:24:31.993 INFO:teuthology.orchestra.run.smithi005.stdout: cluster: 2024-03-22T05:24:31.993 INFO:teuthology.orchestra.run.smithi005.stdout: id: f59f36e2-e80a-11ee-95cd-87774f69a715 2024-03-22T05:24:31.993 INFO:teuthology.orchestra.run.smithi005.stdout: health: HEALTH_OK 2024-03-22T05:24:31.994 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-22T05:24:31.994 INFO:teuthology.orchestra.run.smithi005.stdout: services: 2024-03-22T05:24:31.994 INFO:teuthology.orchestra.run.smithi005.stdout: mon: 3 daemons, quorum a,c,b (age 8m) 2024-03-22T05:24:31.994 INFO:teuthology.orchestra.run.smithi005.stdout: mgr: y(active, since 5m), standbys: x 2024-03-22T05:24:31.994 INFO:teuthology.orchestra.run.smithi005.stdout: osd: 8 osds: 8 up (since 5m), 8 in (since 5m) 2024-03-22T05:24:31.994 INFO:teuthology.orchestra.run.smithi005.stdout: rgw: 2 daemons active (2 hosts, 1 zones) 2024-03-22T05:24:31.994 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-22T05:24:31.994 INFO:teuthology.orchestra.run.smithi005.stdout: data: 2024-03-22T05:24:31.994 INFO:teuthology.orchestra.run.smithi005.stdout: pools: 6 pools, 161 pgs 2024-03-22T05:24:31.994 INFO:teuthology.orchestra.run.smithi005.stdout: objects: 241 objects, 585 KiB 2024-03-22T05:24:31.994 INFO:teuthology.orchestra.run.smithi005.stdout: usage: 76 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:24:31.994 INFO:teuthology.orchestra.run.smithi005.stdout: pgs: 161 active+clean 2024-03-22T05:24:31.994 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-22T05:24:31.994 INFO:teuthology.orchestra.run.smithi005.stdout: io: 2024-03-22T05:24:31.995 INFO:teuthology.orchestra.run.smithi005.stdout: client: 1.2 KiB/s rd, 1 op/s rd, 0 op/s wr 2024-03-22T05:24:31.995 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-22T05:24:32.350 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:24:32 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/2948821199' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-22T05:24:32.350 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:24:32 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/2948821199' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-22T05:24:32.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:24:32 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/2948821199' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-22T05:24:32.633 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:24:32 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:24:32] "GET /metrics HTTP/1.1" 200 207870 "" "Prometheus/2.33.4" 2024-03-22T05:24:32.657 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph orch ls' 2024-03-22T05:24:33.279 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:24:33 smithi005 ceph-mon[24793]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:24:33.279 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:24:33 smithi005 ceph-mon[29523]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:24:33.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:24:33 smithi186 ceph-mon[25720]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:24:33.544 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:24:33 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:24:33] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-22T05:24:34.295 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:24:34 smithi005 ceph-mon[24793]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:24:34.295 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:24:34 smithi005 ceph-mon[29523]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:24:34.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:24:34 smithi186 ceph-mon[25720]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:24:34.597 INFO:teuthology.orchestra.run.smithi005.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-22T05:24:34.597 INFO:teuthology.orchestra.run.smithi005.stdout:alertmanager ?:9093,9094 1/1 3m ago 5m smithi005=a;count:1 2024-03-22T05:24:34.597 INFO:teuthology.orchestra.run.smithi005.stdout:grafana ?:3000 1/1 3m ago 4m smithi186=a;count:1 2024-03-22T05:24:34.597 INFO:teuthology.orchestra.run.smithi005.stdout:iscsi.foo 1/1 3m ago 3m count:1 2024-03-22T05:24:34.597 INFO:teuthology.orchestra.run.smithi005.stdout:mgr 2/2 3m ago 8m smithi005=y;smithi186=x;count:2 2024-03-22T05:24:34.598 INFO:teuthology.orchestra.run.smithi005.stdout:mon 3/3 3m ago 8m smithi005:172.21.15.5=a;smithi005:[v2:172.21.15.5:3301,v1:172.21.15.5:6790]=c;smithi186:172.21.15.186=b;count:3 2024-03-22T05:24:34.598 INFO:teuthology.orchestra.run.smithi005.stdout:node-exporter ?:9100 2/2 3m ago 5m smithi005=a;smithi186=b;count:2 2024-03-22T05:24:34.598 INFO:teuthology.orchestra.run.smithi005.stdout:osd 8 3m ago - 2024-03-22T05:24:34.598 INFO:teuthology.orchestra.run.smithi005.stdout:prometheus ?:9095 1/1 3m ago 5m smithi186=a;count:1 2024-03-22T05:24:34.598 INFO:teuthology.orchestra.run.smithi005.stdout:rgw.foo ?:8000 2/2 3m ago 3m count:2 2024-03-22T05:24:35.207 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph orch daemon redeploy "mgr.$(ceph mgr dump -f json | jq .standbys | jq .[] | jq -r .name)" --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-03-22T05:24:35.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:24:35 smithi005 ceph-mon[24793]: from='client.14973 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:24:35.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:24:35 smithi005 ceph-mon[29523]: from='client.14973 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:24:35.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:24:35 smithi186 ceph-mon[25720]: from='client.14973 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:24:36.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:24:36 smithi005 ceph-mon[24793]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:24:36.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:24:36 smithi005 ceph-mon[24793]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:24:36.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:24:36 smithi005 ceph-mon[29523]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:24:36.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:24:36 smithi005 ceph-mon[29523]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:24:36.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:24:36 smithi186 ceph-mon[25720]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:24:36.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:24:36 smithi186 ceph-mon[25720]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:24:37.369 INFO:teuthology.orchestra.run.smithi005.stdout:Scheduled to redeploy mgr.x on host 'smithi186' 2024-03-22T05:24:37.434 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:24:37 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/2152669062' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-22T05:24:37.434 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:24:37 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/2152669062' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-22T05:24:37.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:24:37 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/2152669062' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-22T05:24:38.088 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph orch ps --refresh' 2024-03-22T05:24:38.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:24:38 smithi005 ceph-mon[24793]: from='client.24898 -' entity='client.admin' cmd=[{"prefix": "orch daemon redeploy", "name": "mgr.x", "image": "quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:24:38.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:24:38 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:24:38.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:24:38 smithi005 ceph-mon[24793]: Schedule redeploy daemon mgr.x 2024-03-22T05:24:38.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:24:38 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:24:38.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:24:38 smithi005 ceph-mon[24793]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:24:38.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:24:38 smithi005 ceph-mon[29523]: from='client.24898 -' entity='client.admin' cmd=[{"prefix": "orch daemon redeploy", "name": "mgr.x", "image": "quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:24:38.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:24:38 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:24:38.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:24:38 smithi005 ceph-mon[29523]: Schedule redeploy daemon mgr.x 2024-03-22T05:24:38.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:24:38 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:24:38.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:24:38 smithi005 ceph-mon[29523]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:24:38.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:24:38 smithi186 ceph-mon[25720]: from='client.24898 -' entity='client.admin' cmd=[{"prefix": "orch daemon redeploy", "name": "mgr.x", "image": "quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:24:38.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:24:38 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:24:38.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:24:38 smithi186 ceph-mon[25720]: Schedule redeploy daemon mgr.x 2024-03-22T05:24:38.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:24:38 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:24:38.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:24:38 smithi186 ceph-mon[25720]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:24:40.532 INFO:teuthology.orchestra.run.smithi005.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T05:24:40.532 INFO:teuthology.orchestra.run.smithi005.stdout:alertmanager.a smithi005 *:9093,9094 running (3m) 3m ago 4m 16.7M - ba2b418f427c ee43e9b1c427 2024-03-22T05:24:40.532 INFO:teuthology.orchestra.run.smithi005.stdout:grafana.a smithi186 *:3000 running (3m) 3m ago 3m 49.6M - 8.3.5 dad864ee21e9 27324e98a8bb 2024-03-22T05:24:40.532 INFO:teuthology.orchestra.run.smithi005.stdout:iscsi.foo.smithi005.mcdwbp smithi005 running (3m) 3m ago 3m 63.6M - 3.5 e1d6a67b021e a2293f952ef5 2024-03-22T05:24:40.532 INFO:teuthology.orchestra.run.smithi005.stdout:mgr.x smithi186 *:8443 running (8m) 3m ago 8m 411M - 17.2.0 e1d6a67b021e febfc27d3c00 2024-03-22T05:24:40.532 INFO:teuthology.orchestra.run.smithi005.stdout:mgr.y smithi005 *:9283 running (10m) 3m ago 10m 454M - 17.2.0 e1d6a67b021e fcee4f81e3ee 2024-03-22T05:24:40.532 INFO:teuthology.orchestra.run.smithi005.stdout:mon.a smithi005 running (10m) 3m ago 10m 47.1M 2048M 17.2.0 e1d6a67b021e 38e0bba4bb03 2024-03-22T05:24:40.532 INFO:teuthology.orchestra.run.smithi005.stdout:mon.b smithi186 running (8m) 3m ago 8m 38.6M 2048M 17.2.0 e1d6a67b021e 323f53f932f6 2024-03-22T05:24:40.533 INFO:teuthology.orchestra.run.smithi005.stdout:mon.c smithi005 running (9m) 3m ago 9m 37.7M 2048M 17.2.0 e1d6a67b021e 00ec24d38612 2024-03-22T05:24:40.533 INFO:teuthology.orchestra.run.smithi005.stdout:node-exporter.a smithi005 *:9100 running (4m) 3m ago 4m 13.3M - 1dbe0e931976 1ee16c40186b 2024-03-22T05:24:40.533 INFO:teuthology.orchestra.run.smithi005.stdout:node-exporter.b smithi186 *:9100 running (4m) 3m ago 4m 17.4M - 1dbe0e931976 e1c552dad930 2024-03-22T05:24:40.533 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0 smithi005 running (8m) 3m ago 8m 47.7M 4017M 17.2.0 e1d6a67b021e ef5c7053dd37 2024-03-22T05:24:40.533 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1 smithi005 running (7m) 3m ago 7m 49.6M 4017M 17.2.0 e1d6a67b021e a1e7ee1017b6 2024-03-22T05:24:40.533 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2 smithi005 running (7m) 3m ago 7m 42.1M 4017M 17.2.0 e1d6a67b021e 462ea63ad0e5 2024-03-22T05:24:40.533 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3 smithi005 running (7m) 3m ago 7m 48.4M 4017M 17.2.0 e1d6a67b021e b25432a7239b 2024-03-22T05:24:40.533 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4 smithi186 running (6m) 3m ago 6m 45.9M 4272M 17.2.0 e1d6a67b021e cd963671f340 2024-03-22T05:24:40.533 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5 smithi186 running (6m) 3m ago 6m 43.7M 4272M 17.2.0 e1d6a67b021e f98c87f1a2cd 2024-03-22T05:24:40.533 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6 smithi186 running (6m) 3m ago 5m 43.5M 4272M 17.2.0 e1d6a67b021e 704ac49c68c4 2024-03-22T05:24:40.533 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7 smithi186 running (5m) 3m ago 5m 45.8M 4272M 17.2.0 e1d6a67b021e f415b90edf63 2024-03-22T05:24:40.533 INFO:teuthology.orchestra.run.smithi005.stdout:prometheus.a smithi186 *:9095 running (3m) 3m ago 4m 42.6M - 514e6a882f6e f520626945ac 2024-03-22T05:24:40.533 INFO:teuthology.orchestra.run.smithi005.stdout:rgw.foo.smithi005.exdhqu smithi005 *:8000 running (3m) 3m ago 3m 72.2M - 17.2.0 e1d6a67b021e 6bb27fe11553 2024-03-22T05:24:40.534 INFO:teuthology.orchestra.run.smithi005.stdout:rgw.foo.smithi186.mcgodk smithi186 *:8000 running (3m) 3m ago 3m 69.7M - 17.2.0 e1d6a67b021e 19b5c20a34f7 2024-03-22T05:24:41.166 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:24:41 smithi005 ceph-mon[24793]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:24:41.166 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:24:41 smithi005 ceph-mon[29523]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:24:41.192 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'sleep 180' 2024-03-22T05:24:41.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:24:41 smithi186 ceph-mon[25720]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:24:42.449 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:24:42 smithi005 ceph-mon[24793]: from='client.14991 -' entity='client.admin' cmd=[{"prefix": "orch ps", "refresh": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:24:42.450 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:24:42 smithi005 ceph-mon[29523]: from='client.14991 -' entity='client.admin' cmd=[{"prefix": "orch ps", "refresh": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:24:42.450 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:24:42 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:24:42] "GET /metrics HTTP/1.1" 200 207870 "" "Prometheus/2.33.4" 2024-03-22T05:24:42.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:24:42 smithi186 ceph-mon[25720]: from='client.14991 -' entity='client.admin' cmd=[{"prefix": "orch ps", "refresh": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:24:43.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:24:43 smithi005 ceph-mon[29523]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:24:43.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:24:43 smithi005 ceph-mon[24793]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:24:43.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:24:43 smithi186 ceph-mon[25720]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:24:43.544 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:24:43 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:24:43] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-22T05:24:44.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:24:44 smithi005 ceph-mon[24793]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:24:44.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:24:44 smithi005 ceph-mon[29523]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:24:44.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:24:44 smithi186 ceph-mon[25720]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:24:47.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:24:47 smithi005 ceph-mon[29523]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:24:47.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:24:47 smithi005 ceph-mon[29523]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:24:47.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:24:47 smithi005 ceph-mon[24793]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:24:47.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:24:47 smithi005 ceph-mon[24793]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:24:47.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:24:47 smithi186 ceph-mon[25720]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:24:47.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:24:47 smithi186 ceph-mon[25720]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:24:49.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:24:49 smithi005 ceph-mon[24793]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:24:49.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:24:49 smithi005 ceph-mon[29523]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:24:49.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:24:49 smithi186 ceph-mon[25720]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:24:51.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:24:51 smithi005 ceph-mon[24793]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:24:51.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:24:51 smithi005 ceph-mon[29523]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:24:51.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:24:51 smithi186 ceph-mon[25720]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:24:52.702 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:24:52 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:24:52] "GET /metrics HTTP/1.1" 200 207876 "" "Prometheus/2.33.4" 2024-03-22T05:24:53.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:24:53 smithi005 ceph-mon[24793]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:24:53.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:24:53 smithi005 ceph-mon[29523]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:24:53.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:24:53 smithi186 ceph-mon[25720]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:24:53.544 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:24:53 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:24:53] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-22T05:24:54.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:24:54 smithi005 ceph-mon[24793]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:24:54.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:24:54 smithi005 ceph-mon[29523]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:24:54.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:24:54 smithi186 ceph-mon[25720]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:24:57.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:24:57 smithi005 ceph-mon[24793]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:24:57.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:24:57 smithi005 ceph-mon[24793]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:24:57.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:24:57 smithi005 ceph-mon[29523]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:24:57.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:24:57 smithi005 ceph-mon[29523]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:24:57.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:24:57 smithi186 ceph-mon[25720]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:24:57.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:24:57 smithi186 ceph-mon[25720]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:24:59.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:24:59 smithi005 ceph-mon[24793]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:24:59.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:24:59 smithi005 ceph-mon[29523]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:24:59.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:24:59 smithi186 ceph-mon[25720]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:25:01.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:25:01 smithi005 ceph-mon[24793]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:25:01.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:25:01 smithi005 ceph-mon[29523]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:25:01.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:25:01 smithi186 ceph-mon[25720]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:25:02.702 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:25:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:25:02] "GET /metrics HTTP/1.1" 200 207877 "" "Prometheus/2.33.4" 2024-03-22T05:25:03.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:25:03 smithi005 ceph-mon[24793]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:25:03.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:25:03 smithi005 ceph-mon[29523]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:25:03.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:25:03 smithi186 ceph-mon[25720]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:25:03.544 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:25:03 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:25:03] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-22T05:25:04.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:25:04 smithi005 ceph-mon[24793]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:25:04.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:25:04 smithi005 ceph-mon[29523]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:25:04.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:25:04 smithi186 ceph-mon[25720]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:25:07.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:25:07 smithi005 ceph-mon[24793]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:25:07.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:25:07 smithi005 ceph-mon[24793]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:25:07.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:25:07 smithi005 ceph-mon[29523]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:25:07.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:25:07 smithi005 ceph-mon[29523]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:25:07.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:25:07 smithi186 ceph-mon[25720]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:25:07.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:25:07 smithi186 ceph-mon[25720]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:25:09.389 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:25:09 smithi005 ceph-mon[24793]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:25:09.390 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:25:09 smithi005 ceph-mon[29523]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:25:09.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:25:09 smithi186 ceph-mon[25720]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:25:11.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:25:11 smithi186 ceph-mon[25720]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:25:11.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:25:11 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:25:11.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:25:11 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:25:11.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:25:11 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:25:11.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:25:11 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:25:11.386 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:25:11 smithi005 ceph-mon[24793]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:25:11.386 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:25:11 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:25:11.386 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:25:11 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:25:11.386 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:25:11 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:25:11.386 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:25:11 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:25:11.386 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:25:11 smithi005 ceph-mon[29523]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:25:11.386 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:25:11 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:25:11.386 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:25:11 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:25:11.386 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:25:11 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:25:11.386 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:25:11 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:25:12.702 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:25:12 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:25:12] "GET /metrics HTTP/1.1" 200 207877 "" "Prometheus/2.33.4" 2024-03-22T05:25:13.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:25:13 smithi005 ceph-mon[24793]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:25:13.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:25:13 smithi005 ceph-mon[29523]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:25:13.435 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:25:13 smithi186 ceph-mon[25720]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:25:13.435 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:25:13 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:25:13] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-22T05:25:15.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:25:14 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:25:15.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:25:14 smithi186 ceph-mon[25720]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:25:15.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:25:14 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:25:15.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:25:14 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:25:15.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:25:14 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-22T05:25:15.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:25:14 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-22T05:25:15.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:25:14 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:25:15.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:25:14 smithi186 ceph-mon[25720]: Deploying daemon mgr.x on smithi186 2024-03-22T05:25:15.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:25:14 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:25:15.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:25:14 smithi005 ceph-mon[24793]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:25:15.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:25:14 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:25:15.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:25:14 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:25:15.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:25:14 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-22T05:25:15.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:25:14 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-22T05:25:15.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:25:14 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:25:15.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:25:14 smithi005 ceph-mon[24793]: Deploying daemon mgr.x on smithi186 2024-03-22T05:25:15.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:25:14 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:25:15.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:25:14 smithi005 ceph-mon[29523]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:25:15.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:25:14 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:25:15.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:25:14 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:25:15.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:25:14 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-22T05:25:15.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:25:14 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-22T05:25:15.204 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:25:14 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:25:15.204 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:25:14 smithi005 ceph-mon[29523]: Deploying daemon mgr.x on smithi186 2024-03-22T05:25:17.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:25:17 smithi005 ceph-mon[24793]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:25:17.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:25:17 smithi005 ceph-mon[24793]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:25:17.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:25:17 smithi005 ceph-mon[29523]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:25:17.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:25:17 smithi005 ceph-mon[29523]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:25:17.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:25:17 smithi186 ceph-mon[25720]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:25:17.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:25:17 smithi186 ceph-mon[25720]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:25:19.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:25:19 smithi005 ceph-mon[24793]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:25:19.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:25:19 smithi005 ceph-mon[29523]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:25:19.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:25:19 smithi186 ceph-mon[25720]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:25:21.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:25:21 smithi005 ceph-mon[24793]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:25:21.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:25:21 smithi005 ceph-mon[29523]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:25:21.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:25:21 smithi186 ceph-mon[25720]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:25:22.702 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:25:22 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:25:22] "GET /metrics HTTP/1.1" 200 207872 "" "Prometheus/2.33.4" 2024-03-22T05:25:23.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:25:23 smithi005 ceph-mon[24793]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:25:23.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:25:23 smithi005 ceph-mon[29523]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:25:23.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:25:23 smithi186 ceph-mon[25720]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:25:23.544 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:25:23 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:25:23] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-22T05:25:23.952 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:25:23 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[49335]: level=warn ts=2024-03-22T05:25:23.508Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.186:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.186 because it doesn't contain any IP SANs" 2024-03-22T05:25:23.952 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:25:23 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[49335]: level=warn ts=2024-03-22T05:25:23.508Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.5:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.5 because it doesn't contain any IP SANs" 2024-03-22T05:25:24.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:25:24 smithi005 ceph-mon[24793]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:25:24.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:25:24 smithi005 ceph-mon[29523]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:25:24.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:25:24 smithi186 ceph-mon[25720]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:25:27.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:25:27 smithi005 ceph-mon[24793]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:25:27.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:25:27 smithi005 ceph-mon[24793]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:25:27.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:25:27 smithi005 ceph-mon[29523]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:25:27.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:25:27 smithi005 ceph-mon[29523]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:25:27.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:25:27 smithi186 ceph-mon[25720]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:25:27.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:25:27 smithi186 ceph-mon[25720]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:25:28.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:25:28 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-22T05:25:28.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:25:28 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-22T05:25:28.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:25:28 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-22T05:25:28.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:25:28 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-22T05:25:28.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:25:28 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-22T05:25:28.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:25:28 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-22T05:25:29.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:25:29 smithi005 ceph-mon[24793]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:25:29.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:25:29 smithi005 ceph-mon[29523]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:25:29.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:25:29 smithi186 ceph-mon[25720]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:25:31.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:25:31 smithi005 ceph-mon[24793]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:25:31.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:25:31 smithi005 ceph-mon[29523]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:25:31.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:25:31 smithi186 ceph-mon[25720]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:25:32.702 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:25:32 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:25:32] "GET /metrics HTTP/1.1" 200 207885 "" "Prometheus/2.33.4" 2024-03-22T05:25:33.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:25:33 smithi005 ceph-mon[24793]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-22T05:25:33.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:25:33 smithi005 ceph-mon[29523]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-22T05:25:33.546 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:25:33 smithi186 ceph-mon[25720]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-22T05:25:33.546 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:25:33 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:25:33] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-22T05:25:33.954 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:25:33 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[49335]: level=error ts=2024-03-22T05:25:33.506Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.186:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.186 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.5:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.5 because it doesn't contain any IP SANs" 2024-03-22T05:25:33.954 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:25:33 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[49335]: level=warn ts=2024-03-22T05:25:33.508Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.186:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.186 because it doesn't contain any IP SANs" 2024-03-22T05:25:33.954 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:25:33 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[49335]: level=warn ts=2024-03-22T05:25:33.508Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.5:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.5 because it doesn't contain any IP SANs" 2024-03-22T05:25:34.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:25:34 smithi005 ceph-mon[24793]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:25:34.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:25:34 smithi005 ceph-mon[29523]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:25:34.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:25:34 smithi186 ceph-mon[25720]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:25:37.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:25:37 smithi005 ceph-mon[24793]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:25:37.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:25:37 smithi005 ceph-mon[24793]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-22T05:25:37.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:25:37 smithi005 ceph-mon[29523]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:25:37.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:25:37 smithi005 ceph-mon[29523]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-22T05:25:37.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:25:37 smithi186 ceph-mon[25720]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:25:37.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:25:37 smithi186 ceph-mon[25720]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-22T05:25:39.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:25:39 smithi005 ceph-mon[24793]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-22T05:25:39.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:25:39 smithi005 ceph-mon[29523]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-22T05:25:39.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:25:39 smithi186 ceph-mon[25720]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-22T05:25:41.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:25:41 smithi005 ceph-mon[24793]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:25:41.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:25:41 smithi005 ceph-mon[29523]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:25:41.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:25:41 smithi186 ceph-mon[25720]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:25:42.702 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:25:42 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:25:42] "GET /metrics HTTP/1.1" 200 207885 "" "Prometheus/2.33.4" 2024-03-22T05:25:43.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:25:43 smithi005 ceph-mon[24793]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:25:43.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:25:43 smithi005 ceph-mon[29523]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:25:43.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:25:43 smithi186 ceph-mon[25720]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:25:43.544 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:25:43 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[26890]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:25:43] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-22T05:25:43.952 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:25:43 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[49335]: level=error ts=2024-03-22T05:25:43.506Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.186:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.186 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.5:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.5 because it doesn't contain any IP SANs" 2024-03-22T05:25:43.952 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:25:43 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[49335]: level=warn ts=2024-03-22T05:25:43.508Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.5:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.5 because it doesn't contain any IP SANs" 2024-03-22T05:25:43.952 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:25:43 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[49335]: level=warn ts=2024-03-22T05:25:43.509Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.186:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.186 because it doesn't contain any IP SANs" 2024-03-22T05:25:44.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:25:44 smithi005 ceph-mon[24793]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:25:44.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:25:44 smithi005 ceph-mon[29523]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:25:44.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:25:44 smithi186 ceph-mon[25720]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:25:46.294 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:25:46 smithi186 systemd[1]: Stopping Ceph mgr.x for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:25:46.294 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:25:46 smithi186 bash[45736]: Error: no container with name or ID "ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr.x" found: no such container 2024-03-22T05:25:46.672 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:25:46 smithi186 podman[45746]: 2024-03-22 05:25:46.351520512 +0000 UTC m=+0.130181868 container died febfc27d3c00bc86b6afb259b6a501db06db4e0a99882d1e3817b3cc8a2e57c4 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, GIT_BRANCH=HEAD, release=754, vcs-type=git, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, GIT_CLEAN=True) 2024-03-22T05:25:47.043 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:25:46 smithi186 podman[45746]: 2024-03-22 05:25:46.864340443 +0000 UTC m=+0.643001806 container cleanup febfc27d3c00bc86b6afb259b6a501db06db4e0a99882d1e3817b3cc8a2e57c4 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, maintainer=Guillaume Abrioux , release=754, io.openshift.tags=base centos centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, name=centos-stream, GIT_BRANCH=HEAD, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, ceph=True, RELEASE=HEAD, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc.) 2024-03-22T05:25:47.044 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:25:46 smithi186 bash[45746]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x 2024-03-22T05:25:47.044 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:25:46 smithi186 bash[45773]: Error: no container with name or ID "ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr.x" found: no such container 2024-03-22T05:25:47.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:25:47 smithi005 ceph-mon[24793]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:25:47.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:25:47 smithi005 ceph-mon[24793]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:25:47.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:25:47 smithi005 ceph-mon[29523]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:25:47.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:25:47 smithi005 ceph-mon[29523]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:25:47.499 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:25:47 smithi186 ceph-mon[25720]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:25:47.500 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:25:47 smithi186 ceph-mon[25720]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:25:47.500 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:25:47 smithi186 podman[45761]: 2024-03-22 05:25:47.099211182 +0000 UTC m=+0.740732940 container remove febfc27d3c00bc86b6afb259b6a501db06db4e0a99882d1e3817b3cc8a2e57c4 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, io.openshift.expose-services=, com.redhat.component=centos-stream-container, name=centos-stream, GIT_CLEAN=True, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, distribution-scope=public, RELEASE=HEAD, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-03-22T05:25:47.500 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:25:47 smithi186 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-03-22T05:25:47.500 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:25:47 smithi186 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@mgr.x.service: Failed with result 'exit-code'. 2024-03-22T05:25:47.500 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:25:47 smithi186 systemd[1]: Stopped Ceph mgr.x for f59f36e2-e80a-11ee-95cd-87774f69a715. 2024-03-22T05:25:47.500 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:25:47 smithi186 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@mgr.x.service: Consumed 17.137s CPU time. 2024-03-22T05:25:47.500 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:25:47 smithi186 systemd[1]: Starting Ceph mgr.x for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:25:47.754 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:25:47 smithi186 podman[45879]: 2024-03-22 05:25:47.500494932 +0000 UTC m=+0.106380960 container create 66924e151e7b8e710dc358650ded703b4f0c25a546259214b18927ca7133ea56 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x, ceph=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-22T05:25:47.754 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:25:47 smithi186 podman[45879]: 2024-03-22 05:25:47.411605598 +0000 UTC m=+0.017491654 image pull aa461e7c5e7f322b169c789b389d594b994ea7ffcbc9e24e563a3ec9b69bc3c1 quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9 2024-03-22T05:25:47.754 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:25:47 smithi186 podman[45879]: 2024-03-22 05:25:47.728484104 +0000 UTC m=+0.334370153 container init 66924e151e7b8e710dc358650ded703b4f0c25a546259214b18927ca7133ea56 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True) 2024-03-22T05:25:47.755 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:25:47 smithi186 podman[45879]: 2024-03-22 05:25:47.730937591 +0000 UTC m=+0.336823620 container start 66924e151e7b8e710dc358650ded703b4f0c25a546259214b18927ca7133ea56 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-22T05:25:48.043 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:25:47 smithi186 ceph-mgr[45898]: -- 172.21.15.186:0/2064169555 <== mon.2 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5582ed86a5a0 con 0x5582ed83e800 2024-03-22T05:25:48.043 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:25:47 smithi186 bash[45879]: 66924e151e7b8e710dc358650ded703b4f0c25a546259214b18927ca7133ea56 2024-03-22T05:25:48.043 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:25:47 smithi186 systemd[1]: Started Ceph mgr.x for f59f36e2-e80a-11ee-95cd-87774f69a715. 2024-03-22T05:25:48.543 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:25:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: 2024-03-22T05:25:48.305+0000 7f040d8e9200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-22T05:25:48.543 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:25:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: 2024-03-22T05:25:48.404+0000 7f040d8e9200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-22T05:25:48.702 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:25:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[49335]: level=warn ts=2024-03-22T05:25:48.323Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=5 err="Post \"https://172.21.15.186:8443/api/prometheus_receiver\": dial tcp 172.21.15.186:8443: connect: connection refused" 2024-03-22T05:25:49.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:25:48 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:25:49.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:25:48 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:25:49.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:25:48 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:25:49.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:25:48 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:25:49.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:25:48 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:25:49.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:25:48 smithi186 ceph-mon[25720]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:25:49.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:25:48 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:25:49.121 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:25:48 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:25:49.121 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:25:48 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:25:49.121 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:25:48 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:25:49.121 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:25:48 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:25:49.121 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:25:48 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:25:49.121 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:25:48 smithi005 ceph-mon[29523]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:25:49.121 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:25:48 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:25:49.122 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:25:48 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:25:49.122 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:25:48 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:25:49.122 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:25:48 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:25:49.122 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:25:48 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:25:49.122 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:25:48 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:25:49.122 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:25:48 smithi005 ceph-mon[24793]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:25:49.122 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:25:48 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:25:49.349 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:25:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: 2024-03-22T05:25:49.091+0000 7f040d8e9200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-22T05:25:49.349 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:25:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: 2024-03-22T05:25:49.197+0000 7f040d8e9200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-22T05:25:49.677 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:25:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: 2024-03-22T05:25:49.349+0000 7f040d8e9200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-22T05:25:49.677 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:25:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: 2024-03-22T05:25:49.450+0000 7f040d8e9200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-22T05:25:49.678 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:25:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: 2024-03-22T05:25:49.540+0000 7f040d8e9200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-22T05:25:49.983 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:25:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: 2024-03-22T05:25:49.827+0000 7f040d8e9200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-22T05:25:49.983 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:25:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: 2024-03-22T05:25:49.983+0000 7f040d8e9200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-22T05:25:50.293 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:25:50 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: 2024-03-22T05:25:50.088+0000 7f040d8e9200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-22T05:25:50.293 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:25:50 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: 2024-03-22T05:25:50.192+0000 7f040d8e9200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-22T05:25:50.793 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:25:50 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: 2024-03-22T05:25:50.379+0000 7f040d8e9200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-22T05:25:51.396 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:25:51 smithi186 ceph-mon[25720]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:25:51.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:25:51 smithi005 ceph-mon[24793]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:25:51.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:25:51 smithi005 ceph-mon[29523]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:25:51.747 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:25:51 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: 2024-03-22T05:25:51.397+0000 7f040d8e9200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-22T05:25:51.747 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:25:51 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: 2024-03-22T05:25:51.546+0000 7f040d8e9200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-22T05:25:52.043 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:25:51 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: 2024-03-22T05:25:51.903+0000 7f040d8e9200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-22T05:25:52.044 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:25:51 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: 2024-03-22T05:25:51.996+0000 7f040d8e9200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-22T05:25:52.543 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:25:52 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: 2024-03-22T05:25:52.197+0000 7f040d8e9200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-22T05:25:52.544 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:25:52 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: 2024-03-22T05:25:52.402+0000 7f040d8e9200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-22T05:25:52.702 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:25:52 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:25:52] "GET /metrics HTTP/1.1" 200 207894 "" "Prometheus/2.33.4" 2024-03-22T05:25:52.964 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:25:52 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: 2024-03-22T05:25:52.704+0000 7f040d8e9200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-22T05:25:53.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:25:53 smithi186 ceph-mon[25720]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:25:53.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:25:53 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:25:53.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:25:53 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:25:53.294 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:25:52 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: 2024-03-22T05:25:52.965+0000 7f040d8e9200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-22T05:25:53.294 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:25:53 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: 2024-03-22T05:25:53.207+0000 7f040d8e9200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-22T05:25:53.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:25:53 smithi005 ceph-mon[24793]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:25:53.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:25:53 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:25:53.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:25:53 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:25:53.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:25:53 smithi005 ceph-mon[29523]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:25:53.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:25:53 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:25:53.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:25:53 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:25:53.952 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:25:53 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[49335]: level=error ts=2024-03-22T05:25:53.507Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.186:8443/api/prometheus_receiver\": dial tcp 172.21.15.186:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.5:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.5 because it doesn't contain any IP SANs" 2024-03-22T05:25:53.952 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:25:53 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[49335]: level=warn ts=2024-03-22T05:25:53.508Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.186:8443/api/prometheus_receiver\": dial tcp 172.21.15.186:8443: connect: connection refused" 2024-03-22T05:25:53.952 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:25:53 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[49335]: level=warn ts=2024-03-22T05:25:53.509Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.5:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.5 because it doesn't contain any IP SANs" 2024-03-22T05:25:54.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:25:54 smithi186 ceph-mon[25720]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:25:54.294 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:25:54 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: 2024-03-22T05:25:54.021+0000 7f040d8e9200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-22T05:25:54.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:25:54 smithi005 ceph-mon[24793]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:25:54.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:25:54 smithi005 ceph-mon[29523]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:25:54.915 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:25:54 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: 2024-03-22T05:25:54.600+0000 7f040d8e9200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-22T05:25:54.915 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:25:54 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: 2024-03-22T05:25:54.697+0000 7f040d8e9200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-22T05:25:54.915 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:25:54 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: 2024-03-22T05:25:54.915+0000 7f040d8e9200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-22T05:25:55.192 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:25:54 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: [22/Mar/2024:05:25:54] ENGINE Bus STARTING 2024-03-22T05:25:55.192 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:25:55 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: CherryPy Checker: 2024-03-22T05:25:55.192 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:25:55 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: The Application mounted at '' has an empty config. 2024-03-22T05:25:55.192 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:25:55 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: 2024-03-22T05:25:55.192 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:25:55 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: [22/Mar/2024:05:25:55] ENGINE Serving on http://:::9283 2024-03-22T05:25:55.193 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:25:55 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: [22/Mar/2024:05:25:55] ENGINE Bus STARTED 2024-03-22T05:25:55.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:25:55 smithi005 ceph-mon[24793]: Standby manager daemon x restarted 2024-03-22T05:25:55.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:25:55 smithi005 ceph-mon[24793]: Standby manager daemon x started 2024-03-22T05:25:55.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:25:55 smithi005 ceph-mon[24793]: from='mgr.? 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-22T05:25:55.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:25:55 smithi005 ceph-mon[24793]: from='mgr.? 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-22T05:25:55.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:25:55 smithi005 ceph-mon[24793]: from='mgr.? 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-22T05:25:55.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:25:55 smithi005 ceph-mon[24793]: from='mgr.? 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-22T05:25:55.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:25:55 smithi005 ceph-mon[29523]: Standby manager daemon x restarted 2024-03-22T05:25:55.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:25:55 smithi005 ceph-mon[29523]: Standby manager daemon x started 2024-03-22T05:25:55.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:25:55 smithi005 ceph-mon[29523]: from='mgr.? 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-22T05:25:55.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:25:55 smithi005 ceph-mon[29523]: from='mgr.? 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-22T05:25:55.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:25:55 smithi005 ceph-mon[29523]: from='mgr.? 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-22T05:25:55.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:25:55 smithi005 ceph-mon[29523]: from='mgr.? 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-22T05:25:55.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:25:55 smithi186 ceph-mon[25720]: Standby manager daemon x restarted 2024-03-22T05:25:55.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:25:55 smithi186 ceph-mon[25720]: Standby manager daemon x started 2024-03-22T05:25:55.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:25:55 smithi186 ceph-mon[25720]: from='mgr.? 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-22T05:25:55.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:25:55 smithi186 ceph-mon[25720]: from='mgr.? 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-22T05:25:55.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:25:55 smithi186 ceph-mon[25720]: from='mgr.? 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-22T05:25:55.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:25:55 smithi186 ceph-mon[25720]: from='mgr.? 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-22T05:25:55.849 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:25:55 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[49335]: level=warn ts=2024-03-22T05:25:55.597Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=4 err="Post \"https://172.21.15.186:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.186 because it doesn't contain any IP SANs" 2024-03-22T05:25:56.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:25:56 smithi186 ceph-mon[25720]: mgrmap e22: y(active, since 6m), standbys: x 2024-03-22T05:25:56.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:25:56 smithi186 ceph-mon[25720]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:25:56.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:25:56 smithi186 ceph-mon[25720]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:25:56.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:25:56 smithi005 ceph-mon[24793]: mgrmap e22: y(active, since 6m), standbys: x 2024-03-22T05:25:56.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:25:56 smithi005 ceph-mon[24793]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:25:56.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:25:56 smithi005 ceph-mon[24793]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:25:56.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:25:56 smithi005 ceph-mon[29523]: mgrmap e22: y(active, since 6m), standbys: x 2024-03-22T05:25:56.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:25:56 smithi005 ceph-mon[29523]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:25:56.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:25:56 smithi005 ceph-mon[29523]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:25:59.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:25:59 smithi005 ceph-mon[29523]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:25:59.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:25:59 smithi005 ceph-mon[24793]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:25:59.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:25:59 smithi186 ceph-mon[25720]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:26:01.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:26:01 smithi005 ceph-mon[29523]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:26:01.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:26:01 smithi005 ceph-mon[24793]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:26:01.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:26:01 smithi186 ceph-mon[25720]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:26:02.447 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:26:02 smithi005 ceph-mon[29523]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:26:02.448 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:26:02 smithi005 ceph-mon[24793]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:26:02.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:26:02 smithi186 ceph-mon[25720]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:26:02.702 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:26:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:26:02] "GET /metrics HTTP/1.1" 200 207891 "" "Prometheus/2.33.4" 2024-03-22T05:26:03.543 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:26:03 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:26:03] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-22T05:26:03.952 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:26:03 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[49335]: level=error ts=2024-03-22T05:26:03.508Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.186:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.186 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.5:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.5 because it doesn't contain any IP SANs" 2024-03-22T05:26:03.952 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:26:03 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[49335]: level=warn ts=2024-03-22T05:26:03.510Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.5:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.5 because it doesn't contain any IP SANs" 2024-03-22T05:26:03.952 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:26:03 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[49335]: level=warn ts=2024-03-22T05:26:03.510Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.186:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.186 because it doesn't contain any IP SANs" 2024-03-22T05:26:04.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:26:04 smithi005 ceph-mon[24793]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:26:04.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:26:04 smithi005 ceph-mon[29523]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:26:04.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:26:04 smithi186 ceph-mon[25720]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:26:07.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:26:07 smithi005 ceph-mon[24793]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:26:07.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:26:07 smithi005 ceph-mon[24793]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:26:07.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:26:07 smithi005 ceph-mon[29523]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:26:07.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:26:07 smithi005 ceph-mon[29523]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:26:07.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:26:07 smithi186 ceph-mon[25720]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:26:07.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:26:07 smithi186 ceph-mon[25720]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:26:09.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:26:09 smithi005 ceph-mon[24793]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:26:09.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:26:09 smithi005 ceph-mon[29523]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:26:09.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:26:09 smithi186 ceph-mon[25720]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:26:10.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:26:10 smithi005 ceph-mon[24793]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:26:10.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:26:10 smithi005 ceph-mon[29523]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:26:10.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:26:10 smithi186 ceph-mon[25720]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:26:12.702 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:26:12 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:26:12] "GET /metrics HTTP/1.1" 200 207891 "" "Prometheus/2.33.4" 2024-03-22T05:26:13.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:26:13 smithi005 ceph-mon[24793]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:26:13.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:26:13 smithi005 ceph-mon[29523]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:26:13.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:26:13 smithi186 ceph-mon[25720]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:26:13.544 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:26:13 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:26:13] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-22T05:26:13.952 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:26:13 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[49335]: level=error ts=2024-03-22T05:26:13.509Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.186:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.186 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.5:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.5 because it doesn't contain any IP SANs" 2024-03-22T05:26:13.952 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:26:13 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[49335]: level=warn ts=2024-03-22T05:26:13.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.5:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.5 because it doesn't contain any IP SANs" 2024-03-22T05:26:13.952 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:26:13 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[49335]: level=warn ts=2024-03-22T05:26:13.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.186:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.186 because it doesn't contain any IP SANs" 2024-03-22T05:26:14.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:26:14 smithi005 ceph-mon[24793]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:26:14.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:26:14 smithi005 ceph-mon[29523]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:26:14.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:26:14 smithi186 ceph-mon[25720]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:26:17.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:26:17 smithi005 ceph-mon[29523]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:26:17.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:26:17 smithi005 ceph-mon[29523]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:26:17.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:26:17 smithi005 ceph-mon[24793]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:26:17.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:26:17 smithi005 ceph-mon[24793]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:26:17.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:26:17 smithi186 ceph-mon[25720]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:26:17.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:26:17 smithi186 ceph-mon[25720]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:26:18.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:26:18 smithi005 ceph-mon[24793]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:26:18.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:26:18 smithi005 ceph-mon[29523]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:26:18.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:26:18 smithi186 ceph-mon[25720]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:26:21.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:26:21 smithi005 ceph-mon[24793]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:26:21.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:26:21 smithi005 ceph-mon[29523]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:26:21.490 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:26:21 smithi186 ceph-mon[25720]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:26:22.447 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:26:22 smithi005 ceph-mon[24793]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:26:22.447 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:26:22 smithi005 ceph-mon[29523]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:26:22.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:26:22 smithi186 ceph-mon[25720]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:26:22.702 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:26:22 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:26:22] "GET /metrics HTTP/1.1" 200 207896 "" "Prometheus/2.33.4" 2024-03-22T05:26:23.543 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:26:23 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:26:23] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-22T05:26:23.952 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:26:23 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[49335]: level=error ts=2024-03-22T05:26:23.510Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.186:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.186 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.5:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.5 because it doesn't contain any IP SANs" 2024-03-22T05:26:23.952 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:26:23 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[49335]: level=warn ts=2024-03-22T05:26:23.512Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.5:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.5 because it doesn't contain any IP SANs" 2024-03-22T05:26:23.952 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:26:23 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[49335]: level=warn ts=2024-03-22T05:26:23.512Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.186:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.186 because it doesn't contain any IP SANs" 2024-03-22T05:26:24.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:26:24 smithi005 ceph-mon[24793]: pgmap v245: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:26:24.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:26:24 smithi005 ceph-mon[29523]: pgmap v245: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:26:24.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:26:24 smithi186 ceph-mon[25720]: pgmap v245: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:26:26.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:26:26 smithi005 ceph-mon[24793]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:26:26.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:26:26 smithi005 ceph-mon[24793]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:26:26.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:26:26 smithi005 ceph-mon[29523]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:26:26.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:26:26 smithi005 ceph-mon[29523]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:26:26.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:26:26 smithi186 ceph-mon[25720]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:26:26.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:26:26 smithi186 ceph-mon[25720]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:26:28.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:26:28 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-22T05:26:28.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:26:28 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-22T05:26:28.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:26:28 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-22T05:26:28.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:26:28 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-22T05:26:28.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:26:28 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-22T05:26:28.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:26:28 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-22T05:26:29.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:26:29 smithi005 ceph-mon[24793]: pgmap v247: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:26:29.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:26:29 smithi005 ceph-mon[29523]: pgmap v247: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:26:29.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:26:29 smithi186 ceph-mon[25720]: pgmap v247: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:26:31.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:26:31 smithi005 ceph-mon[24793]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:26:31.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:26:31 smithi005 ceph-mon[29523]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:26:31.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:26:31 smithi186 ceph-mon[25720]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:26:32.702 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:26:32 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:26:32] "GET /metrics HTTP/1.1" 200 207897 "" "Prometheus/2.33.4" 2024-03-22T05:26:33.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:26:33 smithi005 ceph-mon[29523]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:26:33.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:26:33 smithi005 ceph-mon[24793]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:26:33.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:26:33 smithi186 ceph-mon[25720]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:26:33.544 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:26:33 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:26:33] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-22T05:26:33.952 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:26:33 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[49335]: level=error ts=2024-03-22T05:26:33.511Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.5:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.5 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.186:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.186 because it doesn't contain any IP SANs" 2024-03-22T05:26:33.952 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:26:33 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[49335]: level=warn ts=2024-03-22T05:26:33.513Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.5:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.5 because it doesn't contain any IP SANs" 2024-03-22T05:26:33.952 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:26:33 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[49335]: level=warn ts=2024-03-22T05:26:33.513Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.186:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.186 because it doesn't contain any IP SANs" 2024-03-22T05:26:34.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:26:34 smithi005 ceph-mon[24793]: pgmap v250: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:26:34.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:26:34 smithi005 ceph-mon[29523]: pgmap v250: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:26:34.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:26:34 smithi186 ceph-mon[25720]: pgmap v250: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:26:36.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:26:36 smithi005 ceph-mon[24793]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:26:36.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:26:36 smithi005 ceph-mon[24793]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:26:36.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:26:36 smithi005 ceph-mon[29523]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:26:36.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:26:36 smithi005 ceph-mon[29523]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:26:36.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:26:36 smithi186 ceph-mon[25720]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:26:36.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:26:36 smithi186 ceph-mon[25720]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:26:38.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:26:38 smithi005 ceph-mon[29523]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:26:38.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:26:38 smithi005 ceph-mon[24793]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:26:38.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:26:38 smithi186 ceph-mon[25720]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:26:40.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:26:40 smithi005 ceph-mon[29523]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:26:40.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:26:40 smithi005 ceph-mon[24793]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:26:40.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:26:40 smithi186 ceph-mon[25720]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:26:42.447 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:26:42 smithi005 ceph-mon[24793]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:26:42.447 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:26:42 smithi005 ceph-mon[29523]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:26:42.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:26:42 smithi186 ceph-mon[25720]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:26:42.702 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:26:42 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:26:42] "GET /metrics HTTP/1.1" 200 207897 "" "Prometheus/2.33.4" 2024-03-22T05:26:43.543 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:26:43 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:26:43] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-22T05:26:43.952 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:26:43 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[49335]: level=error ts=2024-03-22T05:26:43.512Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.5:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.5 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.186:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.186 because it doesn't contain any IP SANs" 2024-03-22T05:26:43.952 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:26:43 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[49335]: level=warn ts=2024-03-22T05:26:43.514Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.5:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.5 because it doesn't contain any IP SANs" 2024-03-22T05:26:43.952 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:26:43 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[49335]: level=warn ts=2024-03-22T05:26:43.514Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.186:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.186 because it doesn't contain any IP SANs" 2024-03-22T05:26:44.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:26:44 smithi005 ceph-mon[24793]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:26:44.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:26:44 smithi005 ceph-mon[29523]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:26:44.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:26:44 smithi186 ceph-mon[25720]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:26:46.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:26:46 smithi005 ceph-mon[24793]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:26:46.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:26:46 smithi005 ceph-mon[24793]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:26:46.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:26:46 smithi005 ceph-mon[29523]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:26:46.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:26:46 smithi005 ceph-mon[29523]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:26:46.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:26:46 smithi186 ceph-mon[25720]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:26:46.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:26:46 smithi186 ceph-mon[25720]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:26:48.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:26:48 smithi005 ceph-mon[24793]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:26:48.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:26:48 smithi005 ceph-mon[29523]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:26:48.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:26:48 smithi186 ceph-mon[25720]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:26:50.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:26:50 smithi005 ceph-mon[29523]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:26:50.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:26:50 smithi005 ceph-mon[24793]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:26:50.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:26:50 smithi186 ceph-mon[25720]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:26:52.446 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:26:52 smithi005 ceph-mon[24793]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:26:52.447 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:26:52 smithi005 ceph-mon[29523]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:26:52.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:26:52 smithi186 ceph-mon[25720]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:26:52.702 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:26:52 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:26:52] "GET /metrics HTTP/1.1" 200 207899 "" "Prometheus/2.33.4" 2024-03-22T05:26:53.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:26:53 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:26:53.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:26:53 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:26:53.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:26:53 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:26:53.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:26:53 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:26:53.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:26:53 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:26:53.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:26:53 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:26:53.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:26:53 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:26:53.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:26:53 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:26:53.294 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:26:53 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:26:53] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-22T05:26:53.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:26:53 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:26:53.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:26:53 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:26:53.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:26:53 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:26:53.295 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:26:53 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:26:53.953 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:26:53 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[49335]: level=error ts=2024-03-22T05:26:53.512Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.186:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.186 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.5:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.5 because it doesn't contain any IP SANs" 2024-03-22T05:26:53.953 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:26:53 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[49335]: level=warn ts=2024-03-22T05:26:53.514Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.5:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.5 because it doesn't contain any IP SANs" 2024-03-22T05:26:53.954 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:26:53 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[49335]: level=warn ts=2024-03-22T05:26:53.515Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.186:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.186 because it doesn't contain any IP SANs" 2024-03-22T05:26:54.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:26:54 smithi186 ceph-mon[25720]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:26:54.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:26:54 smithi005 ceph-mon[24793]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:26:54.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:26:54 smithi005 ceph-mon[29523]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:26:56.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:26:56 smithi005 ceph-mon[24793]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:26:56.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:26:56 smithi005 ceph-mon[24793]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:26:56.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:26:56 smithi005 ceph-mon[29523]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:26:56.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:26:56 smithi005 ceph-mon[29523]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:26:56.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:26:56 smithi186 ceph-mon[25720]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:26:56.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:26:56 smithi186 ceph-mon[25720]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:26:58.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:26:58 smithi005 ceph-mon[29523]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:26:58.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:26:58 smithi005 ceph-mon[24793]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:26:58.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:26:58 smithi186 ceph-mon[25720]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:27:00.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:27:00 smithi005 ceph-mon[29523]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:27:00.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:27:00 smithi005 ceph-mon[24793]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:27:00.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:27:00 smithi186 ceph-mon[25720]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:27:02.447 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:27:02 smithi005 ceph-mon[24793]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:27:02.447 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:27:02 smithi005 ceph-mon[29523]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:27:02.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:27:02 smithi186 ceph-mon[25720]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:27:02.702 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:27:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:27:02] "GET /metrics HTTP/1.1" 200 207906 "" "Prometheus/2.33.4" 2024-03-22T05:27:03.543 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:27:03 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:27:03] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-22T05:27:03.952 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:27:03 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[49335]: level=error ts=2024-03-22T05:27:03.514Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.5:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.5 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.186:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.186 because it doesn't contain any IP SANs" 2024-03-22T05:27:03.952 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:27:03 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[49335]: level=warn ts=2024-03-22T05:27:03.516Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.5:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.5 because it doesn't contain any IP SANs" 2024-03-22T05:27:03.952 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:27:03 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[49335]: level=warn ts=2024-03-22T05:27:03.516Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.186:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.186 because it doesn't contain any IP SANs" 2024-03-22T05:27:04.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:27:04 smithi186 ceph-mon[25720]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:27:04.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:27:04 smithi005 ceph-mon[29523]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:27:04.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:27:04 smithi005 ceph-mon[24793]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:27:06.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:27:06 smithi005 ceph-mon[24793]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:27:06.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:27:06 smithi005 ceph-mon[24793]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:27:06.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:27:06 smithi005 ceph-mon[29523]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:27:06.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:27:06 smithi005 ceph-mon[29523]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:27:06.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:27:06 smithi186 ceph-mon[25720]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:27:06.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:27:06 smithi186 ceph-mon[25720]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:27:08.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:27:08 smithi005 ceph-mon[24793]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:27:08.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:27:08 smithi005 ceph-mon[29523]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:27:08.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:27:08 smithi186 ceph-mon[25720]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:27:10.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:27:10 smithi005 ceph-mon[24793]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:27:10.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:27:10 smithi005 ceph-mon[29523]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:27:10.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:27:10 smithi186 ceph-mon[25720]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:27:12.447 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:27:12 smithi005 ceph-mon[24793]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:27:12.447 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:27:12 smithi005 ceph-mon[29523]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:27:12.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:27:12 smithi186 ceph-mon[25720]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:27:12.702 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:27:12 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:27:12] "GET /metrics HTTP/1.1" 200 207906 "" "Prometheus/2.33.4" 2024-03-22T05:27:13.543 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:27:13 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:27:13] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-22T05:27:13.952 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:27:13 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[49335]: level=error ts=2024-03-22T05:27:13.515Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.5:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.5 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.186:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.186 because it doesn't contain any IP SANs" 2024-03-22T05:27:13.952 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:27:13 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[49335]: level=warn ts=2024-03-22T05:27:13.517Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.5:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.5 because it doesn't contain any IP SANs" 2024-03-22T05:27:13.952 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:27:13 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[49335]: level=warn ts=2024-03-22T05:27:13.517Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.186:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.186 because it doesn't contain any IP SANs" 2024-03-22T05:27:14.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:27:14 smithi186 ceph-mon[25720]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:27:14.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:27:14 smithi005 ceph-mon[24793]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:27:14.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:27:14 smithi005 ceph-mon[29523]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:27:16.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:27:16 smithi005 ceph-mon[29523]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:27:16.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:27:16 smithi005 ceph-mon[29523]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:27:16.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:27:16 smithi005 ceph-mon[24793]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:27:16.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:27:16 smithi005 ceph-mon[24793]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:27:16.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:27:16 smithi186 ceph-mon[25720]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:27:16.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:27:16 smithi186 ceph-mon[25720]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:27:18.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:27:18 smithi005 ceph-mon[29523]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:27:18.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:27:18 smithi005 ceph-mon[24793]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:27:18.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:27:18 smithi186 ceph-mon[25720]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:27:20.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:27:20 smithi005 ceph-mon[24793]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:27:20.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:27:20 smithi005 ceph-mon[29523]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:27:20.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:27:20 smithi186 ceph-mon[25720]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:27:22.447 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:27:22 smithi005 ceph-mon[24793]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:27:22.448 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:27:22 smithi005 ceph-mon[29523]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:27:22.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:27:22 smithi186 ceph-mon[25720]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:27:22.702 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:27:22 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:27:22] "GET /metrics HTTP/1.1" 200 207904 "" "Prometheus/2.33.4" 2024-03-22T05:27:23.543 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:27:23 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:27:23] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-22T05:27:23.952 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:27:23 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[49335]: level=error ts=2024-03-22T05:27:23.515Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.5:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.5 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.186:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.186 because it doesn't contain any IP SANs" 2024-03-22T05:27:23.952 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:27:23 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[49335]: level=warn ts=2024-03-22T05:27:23.517Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.5:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.5 because it doesn't contain any IP SANs" 2024-03-22T05:27:23.952 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:27:23 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[49335]: level=warn ts=2024-03-22T05:27:23.518Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.186:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.186 because it doesn't contain any IP SANs" 2024-03-22T05:27:24.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:27:24 smithi186 ceph-mon[25720]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:27:24.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:27:24 smithi005 ceph-mon[24793]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:27:24.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:27:24 smithi005 ceph-mon[29523]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:27:26.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:27:26 smithi005 ceph-mon[29523]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:27:26.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:27:26 smithi005 ceph-mon[29523]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:27:26.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:27:26 smithi005 ceph-mon[24793]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:27:26.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:27:26 smithi005 ceph-mon[24793]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:27:26.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:27:26 smithi186 ceph-mon[25720]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:27:26.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:27:26 smithi186 ceph-mon[25720]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:27:28.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:27:28 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-22T05:27:28.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:27:28 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-22T05:27:28.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:27:28 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-22T05:27:28.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:27:28 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-22T05:27:28.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:27:28 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-22T05:27:28.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:27:28 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-22T05:27:29.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:27:29 smithi005 ceph-mon[24793]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:27:29.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:27:29 smithi005 ceph-mon[29523]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:27:29.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:27:29 smithi186 ceph-mon[25720]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:27:31.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:27:31 smithi005 ceph-mon[24793]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:27:31.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:27:31 smithi005 ceph-mon[29523]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:27:31.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:27:31 smithi186 ceph-mon[25720]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:27:32.702 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:27:32 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:27:32] "GET /metrics HTTP/1.1" 200 207900 "" "Prometheus/2.33.4" 2024-03-22T05:27:33.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:27:33 smithi005 ceph-mon[24793]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:27:33.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:27:33 smithi005 ceph-mon[29523]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:27:33.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:27:33 smithi186 ceph-mon[25720]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:27:33.544 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:27:33 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:27:33] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-22T05:27:33.952 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:27:33 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[49335]: level=error ts=2024-03-22T05:27:33.516Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.186:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.186 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.5:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.5 because it doesn't contain any IP SANs" 2024-03-22T05:27:33.952 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:27:33 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[49335]: level=warn ts=2024-03-22T05:27:33.518Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.5:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.5 because it doesn't contain any IP SANs" 2024-03-22T05:27:33.952 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:27:33 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[49335]: level=warn ts=2024-03-22T05:27:33.518Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.186:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.186 because it doesn't contain any IP SANs" 2024-03-22T05:27:34.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:27:34 smithi186 ceph-mon[25720]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:27:34.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:27:34 smithi005 ceph-mon[24793]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:27:34.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:27:34 smithi005 ceph-mon[29523]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:27:36.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:27:36 smithi005 ceph-mon[24793]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:27:36.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:27:36 smithi005 ceph-mon[24793]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:27:36.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:27:36 smithi005 ceph-mon[29523]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:27:36.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:27:36 smithi005 ceph-mon[29523]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:27:36.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:27:36 smithi186 ceph-mon[25720]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:27:36.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:27:36 smithi186 ceph-mon[25720]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:27:38.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:27:38 smithi005 ceph-mon[29523]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:27:38.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:27:38 smithi005 ceph-mon[24793]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:27:38.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:27:38 smithi186 ceph-mon[25720]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:27:40.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:27:40 smithi005 ceph-mon[24793]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:27:40.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:27:40 smithi005 ceph-mon[29523]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:27:40.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:27:40 smithi186 ceph-mon[25720]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:27:42.448 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:27:42 smithi005 ceph-mon[24793]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:27:42.448 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:27:42 smithi005 ceph-mon[29523]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:27:42.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:27:42 smithi186 ceph-mon[25720]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:27:42.702 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:27:42 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:27:42] "GET /metrics HTTP/1.1" 200 207900 "" "Prometheus/2.33.4" 2024-03-22T05:27:43.543 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:27:43 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:27:43] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-22T05:27:43.702 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:27:43 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[49335]: level=error ts=2024-03-22T05:27:43.516Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.186:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.186 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.5:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.5 because it doesn't contain any IP SANs" 2024-03-22T05:27:43.702 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:27:43 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[49335]: level=warn ts=2024-03-22T05:27:43.519Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.5:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.5 because it doesn't contain any IP SANs" 2024-03-22T05:27:43.702 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:27:43 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[49335]: level=warn ts=2024-03-22T05:27:43.519Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.186:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.186 because it doesn't contain any IP SANs" 2024-03-22T05:27:43.771 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph orch ps' 2024-03-22T05:27:44.428 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:27:44 smithi005 ceph-mon[24793]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:27:44.428 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:27:44 smithi005 ceph-mon[29523]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:27:44.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:27:44 smithi186 ceph-mon[25720]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:27:45.871 INFO:teuthology.orchestra.run.smithi005.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T05:27:45.872 INFO:teuthology.orchestra.run.smithi005.stdout:alertmanager.a smithi005 *:9093,9094 running (6m) 2m ago 7m 28.8M - ba2b418f427c ee43e9b1c427 2024-03-22T05:27:45.872 INFO:teuthology.orchestra.run.smithi005.stdout:grafana.a smithi186 *:3000 running (6m) 113s ago 6m 57.1M - 8.3.5 dad864ee21e9 27324e98a8bb 2024-03-22T05:27:45.872 INFO:teuthology.orchestra.run.smithi005.stdout:iscsi.foo.smithi005.mcdwbp smithi005 running (6m) 2m ago 6m 46.7M - 3.5 e1d6a67b021e a2293f952ef5 2024-03-22T05:27:45.872 INFO:teuthology.orchestra.run.smithi005.stdout:mgr.x smithi186 *:8443,9283 running (118s) 113s ago 11m 67.0M - 19.0.0-2299-gea01217f aa461e7c5e7f 66924e151e7b 2024-03-22T05:27:45.872 INFO:teuthology.orchestra.run.smithi005.stdout:mgr.y smithi005 *:9283 running (13m) 2m ago 13m 460M - 17.2.0 e1d6a67b021e fcee4f81e3ee 2024-03-22T05:27:45.872 INFO:teuthology.orchestra.run.smithi005.stdout:mon.a smithi005 running (13m) 2m ago 13m 50.5M 2048M 17.2.0 e1d6a67b021e 38e0bba4bb03 2024-03-22T05:27:45.872 INFO:teuthology.orchestra.run.smithi005.stdout:mon.b smithi186 running (11m) 113s ago 11m 39.7M 2048M 17.2.0 e1d6a67b021e 323f53f932f6 2024-03-22T05:27:45.872 INFO:teuthology.orchestra.run.smithi005.stdout:mon.c smithi005 running (12m) 2m ago 12m 36.1M 2048M 17.2.0 e1d6a67b021e 00ec24d38612 2024-03-22T05:27:45.872 INFO:teuthology.orchestra.run.smithi005.stdout:node-exporter.a smithi005 *:9100 running (7m) 2m ago 7m 21.0M - 1dbe0e931976 1ee16c40186b 2024-03-22T05:27:45.873 INFO:teuthology.orchestra.run.smithi005.stdout:node-exporter.b smithi186 *:9100 running (7m) 113s ago 7m 17.3M - 1dbe0e931976 e1c552dad930 2024-03-22T05:27:45.873 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0 smithi005 running (11m) 2m ago 11m 49.2M 4017M 17.2.0 e1d6a67b021e ef5c7053dd37 2024-03-22T05:27:45.873 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1 smithi005 running (10m) 2m ago 10m 51.8M 4017M 17.2.0 e1d6a67b021e a1e7ee1017b6 2024-03-22T05:27:45.873 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2 smithi005 running (10m) 2m ago 10m 45.0M 4017M 17.2.0 e1d6a67b021e 462ea63ad0e5 2024-03-22T05:27:45.873 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3 smithi005 running (10m) 2m ago 10m 51.5M 4017M 17.2.0 e1d6a67b021e b25432a7239b 2024-03-22T05:27:45.873 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4 smithi186 running (9m) 113s ago 9m 50.1M 4272M 17.2.0 e1d6a67b021e cd963671f340 2024-03-22T05:27:45.873 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5 smithi186 running (9m) 113s ago 9m 47.1M 4272M 17.2.0 e1d6a67b021e f98c87f1a2cd 2024-03-22T05:27:45.873 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6 smithi186 running (9m) 113s ago 9m 46.6M 4272M 17.2.0 e1d6a67b021e 704ac49c68c4 2024-03-22T05:27:45.873 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7 smithi186 running (8m) 113s ago 8m 48.4M 4272M 17.2.0 e1d6a67b021e f415b90edf63 2024-03-22T05:27:45.874 INFO:teuthology.orchestra.run.smithi005.stdout:prometheus.a smithi186 *:9095 running (6m) 113s ago 7m 66.5M - 514e6a882f6e f520626945ac 2024-03-22T05:27:45.874 INFO:teuthology.orchestra.run.smithi005.stdout:rgw.foo.smithi005.exdhqu smithi005 *:8000 running (6m) 2m ago 6m 73.5M - 17.2.0 e1d6a67b021e 6bb27fe11553 2024-03-22T05:27:45.874 INFO:teuthology.orchestra.run.smithi005.stdout:rgw.foo.smithi186.mcgodk smithi186 *:8000 running (6m) 113s ago 6m 71.2M - 17.2.0 e1d6a67b021e 19b5c20a34f7 2024-03-22T05:27:46.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:27:46 smithi005 ceph-mon[29523]: from='client.15000 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:27:46.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:27:46 smithi005 ceph-mon[29523]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:27:46.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:27:46 smithi005 ceph-mon[29523]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:27:46.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:27:46 smithi005 ceph-mon[24793]: from='client.15000 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:27:46.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:27:46 smithi005 ceph-mon[24793]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:27:46.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:27:46 smithi005 ceph-mon[24793]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:27:46.513 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph versions' 2024-03-22T05:27:46.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:27:46 smithi186 ceph-mon[25720]: from='client.15000 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:27:46.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:27:46 smithi186 ceph-mon[25720]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:27:46.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:27:46 smithi186 ceph-mon[25720]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:27:48.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:27:48 smithi005 ceph-mon[24793]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:27:48.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:27:48 smithi005 ceph-mon[29523]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:27:48.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:27:48 smithi186 ceph-mon[25720]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:27:48.590 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-03-22T05:27:48.590 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": { 2024-03-22T05:27:48.590 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-03-22T05:27:48.590 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:27:48.590 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr": { 2024-03-22T05:27:48.590 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-03-22T05:27:48.590 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev)": 1 2024-03-22T05:27:48.590 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:27:48.591 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": { 2024-03-22T05:27:48.591 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-22T05:27:48.591 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:27:48.591 INFO:teuthology.orchestra.run.smithi005.stdout: "mds": {}, 2024-03-22T05:27:48.591 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw": { 2024-03-22T05:27:48.591 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-22T05:27:48.591 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:27:48.591 INFO:teuthology.orchestra.run.smithi005.stdout: "overall": { 2024-03-22T05:27:48.591 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 14, 2024-03-22T05:27:48.591 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev)": 1 2024-03-22T05:27:48.591 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-03-22T05:27:48.591 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-03-22T05:27:49.243 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph -s' 2024-03-22T05:27:49.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:27:49 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/1544493606' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:27:49.557 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:27:49 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/1544493606' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:27:49.558 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:27:49 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/1544493606' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:27:50.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:27:50 smithi186 ceph-mon[25720]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:27:50.585 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:27:50 smithi005 ceph-mon[29523]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:27:50.585 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:27:50 smithi005 ceph-mon[24793]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:27:51.386 INFO:teuthology.orchestra.run.smithi005.stdout: cluster: 2024-03-22T05:27:51.386 INFO:teuthology.orchestra.run.smithi005.stdout: id: f59f36e2-e80a-11ee-95cd-87774f69a715 2024-03-22T05:27:51.386 INFO:teuthology.orchestra.run.smithi005.stdout: health: HEALTH_OK 2024-03-22T05:27:51.387 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-22T05:27:51.387 INFO:teuthology.orchestra.run.smithi005.stdout: services: 2024-03-22T05:27:51.387 INFO:teuthology.orchestra.run.smithi005.stdout: mon: 3 daemons, quorum a,c,b (age 11m) 2024-03-22T05:27:51.387 INFO:teuthology.orchestra.run.smithi005.stdout: mgr: y(active, since 8m), standbys: x 2024-03-22T05:27:51.387 INFO:teuthology.orchestra.run.smithi005.stdout: osd: 8 osds: 8 up (since 8m), 8 in (since 8m) 2024-03-22T05:27:51.387 INFO:teuthology.orchestra.run.smithi005.stdout: rgw: 2 daemons active (2 hosts, 1 zones) 2024-03-22T05:27:51.387 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-22T05:27:51.387 INFO:teuthology.orchestra.run.smithi005.stdout: data: 2024-03-22T05:27:51.387 INFO:teuthology.orchestra.run.smithi005.stdout: pools: 6 pools, 161 pgs 2024-03-22T05:27:51.387 INFO:teuthology.orchestra.run.smithi005.stdout: objects: 241 objects, 585 KiB 2024-03-22T05:27:51.387 INFO:teuthology.orchestra.run.smithi005.stdout: usage: 76 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:27:51.387 INFO:teuthology.orchestra.run.smithi005.stdout: pgs: 161 active+clean 2024-03-22T05:27:51.387 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-22T05:27:51.387 INFO:teuthology.orchestra.run.smithi005.stdout: io: 2024-03-22T05:27:51.388 INFO:teuthology.orchestra.run.smithi005.stdout: client: 1.2 KiB/s rd, 1 op/s rd, 0 op/s wr 2024-03-22T05:27:51.388 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-22T05:27:51.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:27:51 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/3704213856' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-22T05:27:51.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:27:51 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/3704213856' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-22T05:27:51.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:27:51 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/3704213856' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-22T05:27:52.033 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph health detail' 2024-03-22T05:27:52.595 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:27:52 smithi005 ceph-mon[24793]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:27:52.596 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:27:52 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:27:52] "GET /metrics HTTP/1.1" 200 207899 "" "Prometheus/2.33.4" 2024-03-22T05:27:52.596 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:27:52 smithi005 ceph-mon[29523]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:27:52.695 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:27:52 smithi186 ceph-mon[25720]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:27:53.293 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:27:53 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:27:53] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-22T05:27:53.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:27:53 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:27:53.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:27:53 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:27:53.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:27:53 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:27:53.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:27:53 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:27:53.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:27:53 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:27:53.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:27:53 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:27:53.703 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:27:53 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[49335]: level=error ts=2024-03-22T05:27:53.518Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.5:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.5 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.186:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.186 because it doesn't contain any IP SANs" 2024-03-22T05:27:53.703 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:27:53 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[49335]: level=warn ts=2024-03-22T05:27:53.520Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.5:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.5 because it doesn't contain any IP SANs" 2024-03-22T05:27:53.703 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:27:53 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[49335]: level=warn ts=2024-03-22T05:27:53.520Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.186:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.186 because it doesn't contain any IP SANs" 2024-03-22T05:27:53.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:27:53 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:27:53.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:27:53 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:27:53.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:27:53 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:27:54.245 INFO:teuthology.orchestra.run.smithi005.stdout:HEALTH_OK 2024-03-22T05:27:54.778 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph versions | jq -e '"'"'.mgr | length == 2'"'"'' 2024-03-22T05:27:55.090 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:27:54 smithi005 ceph-mon[24793]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:27:55.091 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:27:54 smithi005 ceph-mon[24793]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:27:55.091 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:27:54 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/2913657581' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T05:27:55.091 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:27:54 smithi005 ceph-mon[29523]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:27:55.091 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:27:54 smithi005 ceph-mon[29523]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:27:55.091 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:27:54 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/2913657581' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T05:27:55.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:27:54 smithi186 ceph-mon[25720]: from='mgr.14433 172.21.15.5:0/2693611046' entity='mgr.y' 2024-03-22T05:27:55.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:27:54 smithi186 ceph-mon[25720]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:27:55.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:27:54 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/2913657581' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T05:27:56.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:27:56 smithi005 ceph-mon[29523]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:27:56.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:27:56 smithi005 ceph-mon[29523]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:27:56.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:27:56 smithi005 ceph-mon[24793]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:27:56.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:27:56 smithi005 ceph-mon[24793]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:27:56.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:27:56 smithi186 ceph-mon[25720]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:27:56.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:27:56 smithi186 ceph-mon[25720]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:27:56.881 INFO:teuthology.orchestra.run.smithi005.stdout:true 2024-03-22T05:27:57.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:27:57 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/2438485134' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:27:57.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:27:57 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/2438485134' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:27:57.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:27:57 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/2438485134' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:27:57.586 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph mgr fail' 2024-03-22T05:27:58.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:27:58 smithi005 ceph-mon[29523]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:27:58.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:27:58 smithi005 ceph-mon[24793]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:27:58.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:27:58 smithi186 ceph-mon[25720]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 76 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:27:59.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:27:59 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/1719566529' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-03-22T05:27:59.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:27:59 smithi005 ceph-mon[24793]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-03-22T05:27:59.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:27:59 smithi005 ceph-mon[24793]: osdmap e94: 8 total, 8 up, 8 in 2024-03-22T05:27:59.952 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:27:59 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:27:59.686+0000 7f1bf0bda700 -1 mgr handle_mgr_map I was active but no longer amignoring --setuser ceph since I am not root 2024-03-22T05:27:59.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:27:59 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: ignoring --setgroup ceph since I am not root 2024-03-22T05:27:59.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:27:59 smithi005 ceph-mgr[25025]: -- 172.21.15.5:0/3454821275 <== mon.0 v2:172.21.15.5:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x559158df2340 con 0x559158dcb000 2024-03-22T05:27:59.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:27:59 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/1719566529' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-03-22T05:27:59.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:27:59 smithi005 ceph-mon[29523]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-03-22T05:27:59.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:27:59 smithi005 ceph-mon[29523]: osdmap e94: 8 total, 8 up, 8 in 2024-03-22T05:28:00.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:27:59 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/1719566529' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-03-22T05:28:00.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:27:59 smithi186 ceph-mon[25720]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-03-22T05:28:00.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:27:59 smithi186 ceph-mon[25720]: osdmap e94: 8 total, 8 up, 8 in 2024-03-22T05:28:00.044 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:27:59 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: [22/Mar/2024:05:27:59] ENGINE Bus STOPPING 2024-03-22T05:28:00.044 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:27:59 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: [22/Mar/2024:05:27:59] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-22T05:28:00.044 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:27:59 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: [22/Mar/2024:05:27:59] ENGINE Bus STOPPED 2024-03-22T05:28:00.044 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:27:59 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: [22/Mar/2024:05:27:59] ENGINE Bus STARTING 2024-03-22T05:28:00.044 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:27:59 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: [22/Mar/2024:05:27:59] ENGINE Serving on http://:::9283 2024-03-22T05:28:00.044 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:27:59 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: [22/Mar/2024:05:27:59] ENGINE Bus STARTED 2024-03-22T05:28:00.313 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:28:00 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:28:00.218+0000 7f490a9c3000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-22T05:28:00.351 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'sleep 180' 2024-03-22T05:28:00.669 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:28:00 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:28:00.312+0000 7f490a9c3000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-22T05:28:00.939 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:00 smithi005 ceph-mon[24793]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-03-22T05:28:00.940 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:00 smithi005 ceph-mon[24793]: mgrmap e23: x(active, starting, since 0.0561943s) 2024-03-22T05:28:00.940 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:00 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-22T05:28:00.940 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:00 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T05:28:00.940 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:00 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T05:28:00.940 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:00 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-22T05:28:00.940 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:00 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T05:28:00.940 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:00 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T05:28:00.940 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:00 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T05:28:00.940 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:00 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T05:28:00.940 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:00 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T05:28:00.940 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:00 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T05:28:00.940 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:00 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-22T05:28:00.941 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:00 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-22T05:28:00.941 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:00 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-22T05:28:00.941 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:00 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-22T05:28:00.941 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:00 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-22T05:28:00.941 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:00 smithi005 ceph-mon[24793]: Manager daemon x is now available 2024-03-22T05:28:00.941 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:00 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:00.941 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:00 smithi005 ceph-mon[24793]: Queued rgw.foo for migration 2024-03-22T05:28:00.941 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:00 smithi005 ceph-mon[24793]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-03-22T05:28:00.941 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:00 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:00.941 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:00 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:28:00.941 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:00 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:28:00.941 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:00 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-22T05:28:00.941 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:00 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-22T05:28:00.941 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:00 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-22T05:28:00.942 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:00 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-22T05:28:00.942 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:00 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:00.942 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:00 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:00.942 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:00 smithi005 ceph-mon[29523]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-03-22T05:28:00.942 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:00 smithi005 ceph-mon[29523]: mgrmap e23: x(active, starting, since 0.0561943s) 2024-03-22T05:28:00.942 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:00 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-22T05:28:00.942 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:00 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T05:28:00.942 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:00 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T05:28:00.942 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:00 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-22T05:28:00.943 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:00 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T05:28:00.943 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:00 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T05:28:00.943 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:00 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T05:28:00.943 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:00 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T05:28:00.943 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:00 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T05:28:00.943 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:00 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T05:28:00.943 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:00 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-22T05:28:00.943 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:00 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-22T05:28:00.943 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:00 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-22T05:28:00.943 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:00 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-22T05:28:00.943 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:00 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-22T05:28:00.943 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:00 smithi005 ceph-mon[29523]: Manager daemon x is now available 2024-03-22T05:28:00.943 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:00 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:00.944 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:00 smithi005 ceph-mon[29523]: Queued rgw.foo for migration 2024-03-22T05:28:00.944 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:00 smithi005 ceph-mon[29523]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-03-22T05:28:00.944 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:00 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:00.944 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:00 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:28:00.944 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:00 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:28:00.944 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:00 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-22T05:28:00.944 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:00 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-22T05:28:00.944 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:00 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-22T05:28:00.944 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:00 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-22T05:28:00.944 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:00 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:00.944 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:00 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:01.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:00 smithi186 ceph-mon[25720]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-03-22T05:28:01.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:00 smithi186 ceph-mon[25720]: mgrmap e23: x(active, starting, since 0.0561943s) 2024-03-22T05:28:01.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:00 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-22T05:28:01.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:00 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T05:28:01.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:00 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T05:28:01.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:00 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-22T05:28:01.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:00 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T05:28:01.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:00 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T05:28:01.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:00 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T05:28:01.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:00 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T05:28:01.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:00 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T05:28:01.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:00 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T05:28:01.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:00 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-22T05:28:01.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:00 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-22T05:28:01.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:00 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-22T05:28:01.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:00 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-22T05:28:01.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:00 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-22T05:28:01.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:00 smithi186 ceph-mon[25720]: Manager daemon x is now available 2024-03-22T05:28:01.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:00 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:01.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:00 smithi186 ceph-mon[25720]: Queued rgw.foo for migration 2024-03-22T05:28:01.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:00 smithi186 ceph-mon[25720]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-03-22T05:28:01.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:00 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:01.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:00 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:28:01.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:00 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:28:01.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:00 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-22T05:28:01.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:00 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-22T05:28:01.046 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:00 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-22T05:28:01.046 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:00 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-22T05:28:01.046 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:00 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:01.046 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:00 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:01.202 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:28:01 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[49335]: level=warn ts=2024-03-22T05:28:01.108Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=6 err="Post \"https://172.21.15.5:8443//api/prometheus_receiver\": dial tcp 172.21.15.5:8443: connect: connection refused" 2024-03-22T05:28:01.202 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:28:00 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:28:00.954+0000 7f490a9c3000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-22T05:28:01.202 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:28:01 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:28:01.047+0000 7f490a9c3000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-22T05:28:01.202 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:28:01 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:28:01.133+0000 7f490a9c3000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-22T05:28:01.570 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:28:01 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:28:01.285+0000 7f490a9c3000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-22T05:28:01.570 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:28:01 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:28:01.370+0000 7f490a9c3000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-22T05:28:01.570 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:28:01 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:28:01.483+0000 7f490a9c3000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-22T05:28:01.570 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:28:01 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:28:01.569+0000 7f490a9c3000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-22T05:28:01.834 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:28:01 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:28:01.741+0000 7f490a9c3000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-22T05:28:01.834 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:28:01 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:28:01.833+0000 7f490a9c3000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-22T05:28:01.834 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:01 smithi005 ceph-mon[24793]: Deploying cephadm binary to smithi005 2024-03-22T05:28:01.835 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:01 smithi005 ceph-mon[24793]: Deploying cephadm binary to smithi186 2024-03-22T05:28:01.835 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:01 smithi005 ceph-mon[24793]: mgrmap e24: x(active, since 1.06809s) 2024-03-22T05:28:01.835 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:01 smithi005 ceph-mon[24793]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:28:01.835 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:01 smithi005 ceph-mon[29523]: Deploying cephadm binary to smithi005 2024-03-22T05:28:01.835 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:01 smithi005 ceph-mon[29523]: Deploying cephadm binary to smithi186 2024-03-22T05:28:01.835 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:01 smithi005 ceph-mon[29523]: mgrmap e24: x(active, since 1.06809s) 2024-03-22T05:28:01.835 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:01 smithi005 ceph-mon[29523]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:28:02.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:01 smithi186 ceph-mon[25720]: Deploying cephadm binary to smithi005 2024-03-22T05:28:02.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:01 smithi186 ceph-mon[25720]: Deploying cephadm binary to smithi186 2024-03-22T05:28:02.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:01 smithi186 ceph-mon[25720]: mgrmap e24: x(active, since 1.06809s) 2024-03-22T05:28:02.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:01 smithi186 ceph-mon[25720]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:28:03.027 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:28:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:28:02.892+0000 7f490a9c3000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-22T05:28:03.027 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:02 smithi005 ceph-mon[29523]: [22/Mar/2024:05:28:01] ENGINE Bus STARTING 2024-03-22T05:28:03.027 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:02 smithi005 ceph-mon[29523]: [22/Mar/2024:05:28:01] ENGINE Serving on https://172.21.15.186:7150 2024-03-22T05:28:03.027 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:02 smithi005 ceph-mon[29523]: [22/Mar/2024:05:28:01] ENGINE Serving on http://172.21.15.186:8765 2024-03-22T05:28:03.027 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:02 smithi005 ceph-mon[29523]: [22/Mar/2024:05:28:01] ENGINE Bus STARTED 2024-03-22T05:28:03.027 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:02 smithi005 ceph-mon[29523]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:28:03.027 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:02 smithi005 ceph-mon[29523]: mgrmap e25: x(active, since 2s) 2024-03-22T05:28:03.028 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:02 smithi005 ceph-mon[24793]: [22/Mar/2024:05:28:01] ENGINE Bus STARTING 2024-03-22T05:28:03.028 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:02 smithi005 ceph-mon[24793]: [22/Mar/2024:05:28:01] ENGINE Serving on https://172.21.15.186:7150 2024-03-22T05:28:03.028 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:02 smithi005 ceph-mon[24793]: [22/Mar/2024:05:28:01] ENGINE Serving on http://172.21.15.186:8765 2024-03-22T05:28:03.028 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:02 smithi005 ceph-mon[24793]: [22/Mar/2024:05:28:01] ENGINE Bus STARTED 2024-03-22T05:28:03.028 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:02 smithi005 ceph-mon[24793]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:28:03.028 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:02 smithi005 ceph-mon[24793]: mgrmap e25: x(active, since 2s) 2024-03-22T05:28:03.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:02 smithi186 ceph-mon[25720]: [22/Mar/2024:05:28:01] ENGINE Bus STARTING 2024-03-22T05:28:03.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:02 smithi186 ceph-mon[25720]: [22/Mar/2024:05:28:01] ENGINE Serving on https://172.21.15.186:7150 2024-03-22T05:28:03.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:02 smithi186 ceph-mon[25720]: [22/Mar/2024:05:28:01] ENGINE Serving on http://172.21.15.186:8765 2024-03-22T05:28:03.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:02 smithi186 ceph-mon[25720]: [22/Mar/2024:05:28:01] ENGINE Bus STARTED 2024-03-22T05:28:03.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:02 smithi186 ceph-mon[25720]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:28:03.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:02 smithi186 ceph-mon[25720]: mgrmap e25: x(active, since 2s) 2024-03-22T05:28:03.317 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:28:03 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:28:03.026+0000 7f490a9c3000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-22T05:28:03.317 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:28:03 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:28:03.226+0000 7f490a9c3000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-22T05:28:03.543 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:03 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:28:03] "GET /metrics HTTP/1.1" 200 34885 "" "Prometheus/2.33.4" 2024-03-22T05:28:03.702 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:28:03 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:28:03.316+0000 7f490a9c3000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-22T05:28:03.702 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:28:03 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:28:03.457+0000 7f490a9c3000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-22T05:28:03.703 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:28:03 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[49335]: level=error ts=2024-03-22T05:28:03.518Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.186:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.186 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.5:8443//api/prometheus_receiver\": dial tcp 172.21.15.5:8443: connect: connection refused" 2024-03-22T05:28:03.703 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:28:03 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[49335]: level=warn ts=2024-03-22T05:28:03.519Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.5:8443//api/prometheus_receiver\": dial tcp 172.21.15.5:8443: connect: connection refused" 2024-03-22T05:28:03.703 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:28:03 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[49335]: level=warn ts=2024-03-22T05:28:03.521Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.186:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.186 because it doesn't contain any IP SANs" 2024-03-22T05:28:04.103 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:28:03 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:28:03.783+0000 7f490a9c3000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-22T05:28:04.103 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:28:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:28:04.013+0000 7f490a9c3000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-22T05:28:04.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:04 smithi005 ceph-mon[24793]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:28:04.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:04 smithi005 ceph-mon[29523]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:28:04.452 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:28:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:28:04.102+0000 7f490a9c3000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-22T05:28:04.453 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:28:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:28:04.281+0000 7f490a9c3000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-22T05:28:04.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:04 smithi186 ceph-mon[25720]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:28:05.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:05 smithi186 ceph-mon[25720]: mgrmap e26: x(active, since 4s) 2024-03-22T05:28:05.574 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:28:05 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:28:05.246+0000 7f490a9c3000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-22T05:28:05.574 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:05 smithi005 ceph-mon[24793]: mgrmap e26: x(active, since 4s) 2024-03-22T05:28:05.574 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:05 smithi005 ceph-mon[29523]: mgrmap e26: x(active, since 4s) 2024-03-22T05:28:05.829 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:28:05 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:28:05.573+0000 7f490a9c3000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-22T05:28:05.829 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:28:05 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:28:05.675+0000 7f490a9c3000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-22T05:28:06.202 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:28:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[49335]: level=warn ts=2024-03-22T05:28:06.024Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=4 err="Post \"https://172.21.15.5:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.5 because it doesn't contain any IP SANs" 2024-03-22T05:28:06.202 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:28:05 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:28:05.839+0000 7f490a9c3000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-22T05:28:06.202 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:28:05 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: [22/Mar/2024:05:28:05] ENGINE Bus STARTING 2024-03-22T05:28:06.202 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:28:05 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: CherryPy Checker: 2024-03-22T05:28:06.203 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:28:05 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: The Application mounted at '' has an empty config. 2024-03-22T05:28:06.203 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:28:05 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: 2024-03-22T05:28:06.203 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:28:05 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: [22/Mar/2024:05:28:05] ENGINE Serving on http://:::9283 2024-03-22T05:28:06.203 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:28:05 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: [22/Mar/2024:05:28:05] ENGINE Bus STARTED 2024-03-22T05:28:06.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:06 smithi186 ceph-mon[25720]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:28:06.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:06 smithi186 ceph-mon[25720]: Standby manager daemon y started 2024-03-22T05:28:06.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:06 smithi186 ceph-mon[25720]: from='mgr.? 172.21.15.5:0/715956538' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-22T05:28:06.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:06 smithi186 ceph-mon[25720]: from='mgr.? 172.21.15.5:0/715956538' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-22T05:28:06.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:06 smithi186 ceph-mon[25720]: from='mgr.? 172.21.15.5:0/715956538' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-22T05:28:06.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:06 smithi186 ceph-mon[25720]: from='mgr.? 172.21.15.5:0/715956538' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-22T05:28:06.627 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:06 smithi005 ceph-mon[24793]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:28:06.628 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:06 smithi005 ceph-mon[24793]: Standby manager daemon y started 2024-03-22T05:28:06.628 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:06 smithi005 ceph-mon[24793]: from='mgr.? 172.21.15.5:0/715956538' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-22T05:28:06.628 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:06 smithi005 ceph-mon[24793]: from='mgr.? 172.21.15.5:0/715956538' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-22T05:28:06.628 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:06 smithi005 ceph-mon[24793]: from='mgr.? 172.21.15.5:0/715956538' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-22T05:28:06.628 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:06 smithi005 ceph-mon[24793]: from='mgr.? 172.21.15.5:0/715956538' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-22T05:28:06.628 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:06 smithi005 ceph-mon[29523]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:28:06.628 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:06 smithi005 ceph-mon[29523]: Standby manager daemon y started 2024-03-22T05:28:06.628 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:06 smithi005 ceph-mon[29523]: from='mgr.? 172.21.15.5:0/715956538' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-22T05:28:06.628 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:06 smithi005 ceph-mon[29523]: from='mgr.? 172.21.15.5:0/715956538' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-22T05:28:06.628 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:06 smithi005 ceph-mon[29523]: from='mgr.? 172.21.15.5:0/715956538' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-22T05:28:06.628 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:06 smithi005 ceph-mon[29523]: from='mgr.? 172.21.15.5:0/715956538' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-22T05:28:07.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:07 smithi186 ceph-mon[25720]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:28:07.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:07 smithi186 ceph-mon[25720]: mgrmap e27: x(active, since 6s), standbys: y 2024-03-22T05:28:07.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:07 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-22T05:28:07.548 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:07 smithi005 ceph-mon[24793]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:28:07.548 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:07 smithi005 ceph-mon[24793]: mgrmap e27: x(active, since 6s), standbys: y 2024-03-22T05:28:07.548 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:07 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-22T05:28:07.549 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:07 smithi005 ceph-mon[29523]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:28:07.549 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:07 smithi005 ceph-mon[29523]: mgrmap e27: x(active, since 6s), standbys: y 2024-03-22T05:28:07.549 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:07 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-22T05:28:08.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:08 smithi005 ceph-mon[24793]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-22T05:28:08.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:08 smithi005 ceph-mon[29523]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-22T05:28:08.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:08 smithi186 ceph-mon[25720]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-22T05:28:09.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:09 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:09.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:09 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:09.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:09 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:09.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:09 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:09.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:09 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:09.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:09 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:09.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:09 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:09.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:09 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:10.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:09 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:10.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:09 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:10.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:09 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:10.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:09 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:11.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:10 smithi186 ceph-mon[25720]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-22T05:28:11.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:10 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:11.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:10 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:11.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:10 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:28:11.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:10 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:28:11.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:10 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:28:11.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:10 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:28:11.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:10 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:28:11.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:10 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:28:11.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:10 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:28:11.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:10 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:28:11.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:10 smithi186 ceph-mon[25720]: Adjusting osd_memory_target on smithi005 to 2993M 2024-03-22T05:28:11.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:10 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:11.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:10 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:11.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:10 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:11.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:10 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:28:11.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:10 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:28:11.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:10 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:28:11.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:10 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:28:11.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:10 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:28:11.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:10 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:28:11.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:10 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:28:11.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:10 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:28:11.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:10 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:11.046 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:10 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:28:11.046 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:10 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:28:11.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:10 smithi005 ceph-mon[24793]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-22T05:28:11.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:10 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:11.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:10 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:11.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:10 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:28:11.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:10 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:28:11.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:10 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:28:11.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:10 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:28:11.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:10 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:28:11.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:10 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:28:11.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:10 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:28:11.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:10 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:28:11.204 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:10 smithi005 ceph-mon[24793]: Adjusting osd_memory_target on smithi005 to 2993M 2024-03-22T05:28:11.204 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:10 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:11.204 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:10 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:11.204 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:10 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:11.204 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:10 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:28:11.204 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:10 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:28:11.204 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:10 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:28:11.204 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:10 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:28:11.204 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:10 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:28:11.204 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:10 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:28:11.204 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:10 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:28:11.204 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:10 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:28:11.204 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:10 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:11.204 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:10 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:28:11.205 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:10 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:28:11.205 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:10 smithi005 ceph-mon[29523]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-22T05:28:11.205 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:10 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:11.205 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:10 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:11.205 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:10 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:28:11.205 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:10 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:28:11.205 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:10 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:28:11.205 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:10 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:28:11.205 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:10 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:28:11.205 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:10 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:28:11.205 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:10 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:28:11.206 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:10 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:28:11.206 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:10 smithi005 ceph-mon[29523]: Adjusting osd_memory_target on smithi005 to 2993M 2024-03-22T05:28:11.206 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:10 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:11.206 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:10 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:11.206 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:10 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:11.206 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:10 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:28:11.206 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:10 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:28:11.206 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:10 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:28:11.206 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:10 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:28:11.206 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:10 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:28:11.206 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:10 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:28:11.206 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:10 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:28:11.206 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:10 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-22T05:28:11.206 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:10 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:11.207 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:10 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:28:11.207 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:10 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:28:12.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:11 smithi186 ceph-mon[25720]: Adjusting osd_memory_target on smithi186 to 3248M 2024-03-22T05:28:12.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:11 smithi186 ceph-mon[25720]: Updating smithi005:/etc/ceph/ceph.conf 2024-03-22T05:28:12.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:11 smithi186 ceph-mon[25720]: Updating smithi186:/etc/ceph/ceph.conf 2024-03-22T05:28:12.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:11 smithi186 ceph-mon[25720]: Updating smithi186:/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T05:28:12.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:11 smithi186 ceph-mon[25720]: Updating smithi005:/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T05:28:12.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:11 smithi186 ceph-mon[25720]: Updating smithi186:/etc/ceph/ceph.client.admin.keyring 2024-03-22T05:28:12.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:11 smithi186 ceph-mon[25720]: Updating smithi005:/etc/ceph/ceph.client.admin.keyring 2024-03-22T05:28:12.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:11 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:12.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:11 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:12.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:11 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:12.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:11 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:12.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:11 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:12.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:11 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:12.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:11 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:12.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:11 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:12.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:11 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:12.168 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:28:11 smithi005 systemd[1]: Stopping Ceph node-exporter.a for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:28:12.168 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:28:11 smithi005 bash[58511]: Error: no container with name or ID "ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter.a" found: no such container 2024-03-22T05:28:12.168 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:28:11 smithi005 podman[58522]: 2024-03-22 05:28:11.885814117 +0000 UTC m=+0.103177788 container died 1ee16c40186bbe0f00ec98516f41d9f19078571cbf8f93f9ab51c9f0103b574e (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-22T05:28:12.169 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:11 smithi005 ceph-mon[24793]: Adjusting osd_memory_target on smithi186 to 3248M 2024-03-22T05:28:12.169 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:11 smithi005 ceph-mon[24793]: Updating smithi005:/etc/ceph/ceph.conf 2024-03-22T05:28:12.169 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:11 smithi005 ceph-mon[24793]: Updating smithi186:/etc/ceph/ceph.conf 2024-03-22T05:28:12.169 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:11 smithi005 ceph-mon[24793]: Updating smithi186:/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T05:28:12.169 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:11 smithi005 ceph-mon[24793]: Updating smithi005:/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T05:28:12.169 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:11 smithi005 ceph-mon[24793]: Updating smithi186:/etc/ceph/ceph.client.admin.keyring 2024-03-22T05:28:12.169 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:11 smithi005 ceph-mon[24793]: Updating smithi005:/etc/ceph/ceph.client.admin.keyring 2024-03-22T05:28:12.169 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:11 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:12.169 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:11 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:12.169 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:11 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:12.169 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:11 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:12.170 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:11 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:12.170 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:11 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:12.170 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:11 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:12.170 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:11 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:12.170 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:11 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:12.170 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:11 smithi005 ceph-mon[29523]: Adjusting osd_memory_target on smithi186 to 3248M 2024-03-22T05:28:12.170 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:11 smithi005 ceph-mon[29523]: Updating smithi005:/etc/ceph/ceph.conf 2024-03-22T05:28:12.170 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:11 smithi005 ceph-mon[29523]: Updating smithi186:/etc/ceph/ceph.conf 2024-03-22T05:28:12.170 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:11 smithi005 ceph-mon[29523]: Updating smithi186:/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T05:28:12.170 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:11 smithi005 ceph-mon[29523]: Updating smithi005:/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T05:28:12.170 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:11 smithi005 ceph-mon[29523]: Updating smithi186:/etc/ceph/ceph.client.admin.keyring 2024-03-22T05:28:12.170 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:11 smithi005 ceph-mon[29523]: Updating smithi005:/etc/ceph/ceph.client.admin.keyring 2024-03-22T05:28:12.171 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:11 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:12.171 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:11 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:12.171 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:11 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:12.171 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:11 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:12.171 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:11 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:12.171 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:11 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:12.171 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:11 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:12.171 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:11 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:12.171 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:11 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:12.447 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:28:12 smithi005 podman[58522]: 2024-03-22 05:28:12.267658951 +0000 UTC m=+0.485022620 container cleanup 1ee16c40186bbe0f00ec98516f41d9f19078571cbf8f93f9ab51c9f0103b574e (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-22T05:28:12.447 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:28:12 smithi005 bash[58522]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a 2024-03-22T05:28:12.448 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:28:12 smithi005 bash[58548]: Error: no container with name or ID "ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter.a" found: no such container 2024-03-22T05:28:12.702 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:28:12 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:28:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-22T05:28:12.703 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:28:12 smithi005 podman[58536]: 2024-03-22 05:28:12.493718435 +0000 UTC m=+0.609337087 container remove 1ee16c40186bbe0f00ec98516f41d9f19078571cbf8f93f9ab51c9f0103b574e (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-22T05:28:12.703 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:28:12 smithi005 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-03-22T05:28:12.703 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:28:12 smithi005 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-03-22T05:28:12.703 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:28:12 smithi005 systemd[1]: Stopped Ceph node-exporter.a for f59f36e2-e80a-11ee-95cd-87774f69a715. 2024-03-22T05:28:12.703 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:28:12 smithi005 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@node-exporter.a.service: Consumed 3.680s CPU time. 2024-03-22T05:28:12.703 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:28:12 smithi005 systemd[1]: Starting Ceph node-exporter.a for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:28:13.013 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:12 smithi005 ceph-mon[24793]: Updating smithi186:/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-22T05:28:13.013 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:12 smithi005 ceph-mon[24793]: Updating smithi005:/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-22T05:28:13.013 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:12 smithi005 ceph-mon[24793]: Reconfiguring node-exporter.a (dependencies changed)... 2024-03-22T05:28:13.013 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:12 smithi005 ceph-mon[24793]: Reconfiguring daemon node-exporter.a on smithi005 2024-03-22T05:28:13.013 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:12 smithi005 ceph-mon[24793]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-22T05:28:13.013 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:12 smithi005 ceph-mon[29523]: Updating smithi186:/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-22T05:28:13.013 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:12 smithi005 ceph-mon[29523]: Updating smithi005:/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-22T05:28:13.013 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:12 smithi005 ceph-mon[29523]: Reconfiguring node-exporter.a (dependencies changed)... 2024-03-22T05:28:13.014 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:12 smithi005 ceph-mon[29523]: Reconfiguring daemon node-exporter.a on smithi005 2024-03-22T05:28:13.014 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:12 smithi005 ceph-mon[29523]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-22T05:28:13.014 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:28:12 smithi005 podman[58616]: 2024-03-22 05:28:12.631086167 +0000 UTC m=+0.017804353 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-03-22T05:28:13.014 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:28:12 smithi005 podman[58616]: 2024-03-22 05:28:12.744830204 +0000 UTC m=+0.131548413 container create a9085d6042eb06fd812007f7f7bf20615c23c8f5be96a5908548bb8b5139cbc1 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-22T05:28:13.014 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:28:12 smithi005 podman[58616]: 2024-03-22 05:28:12.938727919 +0000 UTC m=+0.325446094 container init a9085d6042eb06fd812007f7f7bf20615c23c8f5be96a5908548bb8b5139cbc1 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-22T05:28:13.015 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:28:12 smithi005 podman[58616]: 2024-03-22 05:28:12.942014527 +0000 UTC m=+0.328732696 container start a9085d6042eb06fd812007f7f7bf20615c23c8f5be96a5908548bb8b5139cbc1 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-22T05:28:13.015 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:28:12 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[58631]: ts=2024-03-22T05:28:12.945Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-03-22T05:28:13.015 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:28:12 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[58631]: ts=2024-03-22T05:28:12.945Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-03-22T05:28:13.015 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:28:12 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[58631]: ts=2024-03-22T05:28:12.945Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-03-22T05:28:13.015 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:28:12 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[58631]: ts=2024-03-22T05:28:12.945Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-22T05:28:13.015 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:28:12 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[58631]: ts=2024-03-22T05:28:12.946Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-03-22T05:28:13.015 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:28:12 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[58631]: ts=2024-03-22T05:28:12.946Z caller=node_exporter.go:115 level=info collector=arp 2024-03-22T05:28:13.015 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:28:12 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[58631]: ts=2024-03-22T05:28:12.946Z caller=node_exporter.go:115 level=info collector=bcache 2024-03-22T05:28:13.015 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:28:12 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[58631]: ts=2024-03-22T05:28:12.946Z caller=node_exporter.go:115 level=info collector=bonding 2024-03-22T05:28:13.015 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:28:12 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[58631]: ts=2024-03-22T05:28:12.946Z caller=node_exporter.go:115 level=info collector=btrfs 2024-03-22T05:28:13.015 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:28:12 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[58631]: ts=2024-03-22T05:28:12.946Z caller=node_exporter.go:115 level=info collector=conntrack 2024-03-22T05:28:13.015 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:28:12 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[58631]: ts=2024-03-22T05:28:12.946Z caller=node_exporter.go:115 level=info collector=cpu 2024-03-22T05:28:13.015 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:28:12 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[58631]: ts=2024-03-22T05:28:12.946Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-03-22T05:28:13.015 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:28:12 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[58631]: ts=2024-03-22T05:28:12.946Z caller=node_exporter.go:115 level=info collector=diskstats 2024-03-22T05:28:13.016 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:28:12 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[58631]: ts=2024-03-22T05:28:12.946Z caller=node_exporter.go:115 level=info collector=dmi 2024-03-22T05:28:13.016 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:28:12 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[58631]: ts=2024-03-22T05:28:12.946Z caller=node_exporter.go:115 level=info collector=edac 2024-03-22T05:28:13.016 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:28:12 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[58631]: ts=2024-03-22T05:28:12.946Z caller=node_exporter.go:115 level=info collector=entropy 2024-03-22T05:28:13.016 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:28:12 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[58631]: ts=2024-03-22T05:28:12.946Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-03-22T05:28:13.016 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:28:12 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[58631]: ts=2024-03-22T05:28:12.946Z caller=node_exporter.go:115 level=info collector=filefd 2024-03-22T05:28:13.016 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:28:12 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[58631]: ts=2024-03-22T05:28:12.946Z caller=node_exporter.go:115 level=info collector=filesystem 2024-03-22T05:28:13.016 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:28:12 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[58631]: ts=2024-03-22T05:28:12.946Z caller=node_exporter.go:115 level=info collector=hwmon 2024-03-22T05:28:13.016 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:28:12 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[58631]: ts=2024-03-22T05:28:12.946Z caller=node_exporter.go:115 level=info collector=infiniband 2024-03-22T05:28:13.016 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:28:12 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[58631]: ts=2024-03-22T05:28:12.946Z caller=node_exporter.go:115 level=info collector=ipvs 2024-03-22T05:28:13.016 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:28:12 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[58631]: ts=2024-03-22T05:28:12.946Z caller=node_exporter.go:115 level=info collector=loadavg 2024-03-22T05:28:13.016 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:28:12 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[58631]: ts=2024-03-22T05:28:12.946Z caller=node_exporter.go:115 level=info collector=mdadm 2024-03-22T05:28:13.016 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:28:12 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[58631]: ts=2024-03-22T05:28:12.946Z caller=node_exporter.go:115 level=info collector=meminfo 2024-03-22T05:28:13.016 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:28:12 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[58631]: ts=2024-03-22T05:28:12.946Z caller=node_exporter.go:115 level=info collector=netclass 2024-03-22T05:28:13.017 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:28:12 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[58631]: ts=2024-03-22T05:28:12.946Z caller=node_exporter.go:115 level=info collector=netdev 2024-03-22T05:28:13.017 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:28:12 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[58631]: ts=2024-03-22T05:28:12.946Z caller=node_exporter.go:115 level=info collector=netstat 2024-03-22T05:28:13.017 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:28:12 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[58631]: ts=2024-03-22T05:28:12.946Z caller=node_exporter.go:115 level=info collector=nfs 2024-03-22T05:28:13.017 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:28:12 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[58631]: ts=2024-03-22T05:28:12.946Z caller=node_exporter.go:115 level=info collector=nfsd 2024-03-22T05:28:13.017 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:28:12 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[58631]: ts=2024-03-22T05:28:12.946Z caller=node_exporter.go:115 level=info collector=nvme 2024-03-22T05:28:13.017 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:28:12 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[58631]: ts=2024-03-22T05:28:12.946Z caller=node_exporter.go:115 level=info collector=os 2024-03-22T05:28:13.017 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:28:12 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[58631]: ts=2024-03-22T05:28:12.946Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-03-22T05:28:13.017 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:28:12 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[58631]: ts=2024-03-22T05:28:12.946Z caller=node_exporter.go:115 level=info collector=pressure 2024-03-22T05:28:13.017 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:28:12 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[58631]: ts=2024-03-22T05:28:12.946Z caller=node_exporter.go:115 level=info collector=rapl 2024-03-22T05:28:13.017 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:28:12 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[58631]: ts=2024-03-22T05:28:12.946Z caller=node_exporter.go:115 level=info collector=schedstat 2024-03-22T05:28:13.018 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:28:12 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[58631]: ts=2024-03-22T05:28:12.946Z caller=node_exporter.go:115 level=info collector=sockstat 2024-03-22T05:28:13.018 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:28:12 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[58631]: ts=2024-03-22T05:28:12.946Z caller=node_exporter.go:115 level=info collector=softnet 2024-03-22T05:28:13.018 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:28:12 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[58631]: ts=2024-03-22T05:28:12.946Z caller=node_exporter.go:115 level=info collector=stat 2024-03-22T05:28:13.018 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:28:12 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[58631]: ts=2024-03-22T05:28:12.946Z caller=node_exporter.go:115 level=info collector=tapestats 2024-03-22T05:28:13.018 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:28:12 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[58631]: ts=2024-03-22T05:28:12.946Z caller=node_exporter.go:115 level=info collector=textfile 2024-03-22T05:28:13.018 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:28:12 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[58631]: ts=2024-03-22T05:28:12.946Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-03-22T05:28:13.018 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:28:12 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[58631]: ts=2024-03-22T05:28:12.946Z caller=node_exporter.go:115 level=info collector=time 2024-03-22T05:28:13.018 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:28:12 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[58631]: ts=2024-03-22T05:28:12.946Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-03-22T05:28:13.018 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:28:12 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[58631]: ts=2024-03-22T05:28:12.946Z caller=node_exporter.go:115 level=info collector=uname 2024-03-22T05:28:13.018 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:28:12 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[58631]: ts=2024-03-22T05:28:12.946Z caller=node_exporter.go:115 level=info collector=vmstat 2024-03-22T05:28:13.018 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:28:12 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[58631]: ts=2024-03-22T05:28:12.946Z caller=node_exporter.go:115 level=info collector=xfs 2024-03-22T05:28:13.018 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:28:12 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[58631]: ts=2024-03-22T05:28:12.946Z caller=node_exporter.go:115 level=info collector=zfs 2024-03-22T05:28:13.018 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:28:12 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[58631]: ts=2024-03-22T05:28:12.946Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-03-22T05:28:13.019 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:28:12 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[58631]: ts=2024-03-22T05:28:12.946Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-03-22T05:28:13.019 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:28:13 smithi005 bash[58616]: a9085d6042eb06fd812007f7f7bf20615c23c8f5be96a5908548bb8b5139cbc1 2024-03-22T05:28:13.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:12 smithi186 ceph-mon[25720]: Updating smithi186:/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-22T05:28:13.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:12 smithi186 ceph-mon[25720]: Updating smithi005:/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-22T05:28:13.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:12 smithi186 ceph-mon[25720]: Reconfiguring node-exporter.a (dependencies changed)... 2024-03-22T05:28:13.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:12 smithi186 ceph-mon[25720]: Reconfiguring daemon node-exporter.a on smithi005 2024-03-22T05:28:13.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:12 smithi186 ceph-mon[25720]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-22T05:28:13.452 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:28:13 smithi005 systemd[1]: Started Ceph node-exporter.a for f59f36e2-e80a-11ee-95cd-87774f69a715. 2024-03-22T05:28:13.543 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:13 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:28:13] "GET /metrics HTTP/1.1" 200 34885 "" "Prometheus/2.33.4" 2024-03-22T05:28:13.952 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:28:13 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[49335]: level=error ts=2024-03-22T05:28:13.519Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.186:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.186 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.5:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.5 because it doesn't contain any IP SANs" 2024-03-22T05:28:13.952 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:28:13 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[49335]: level=warn ts=2024-03-22T05:28:13.521Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.5:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.5 because it doesn't contain any IP SANs" 2024-03-22T05:28:13.952 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:28:13 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[49335]: level=warn ts=2024-03-22T05:28:13.521Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.186:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.186 because it doesn't contain any IP SANs" 2024-03-22T05:28:14.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:14 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:14.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:14 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:14.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:14 smithi005 ceph-mon[24793]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-22T05:28:14.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:14 smithi005 ceph-mon[24793]: Reconfiguring daemon alertmanager.a on smithi005 2024-03-22T05:28:14.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:14 smithi005 ceph-mon[24793]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-22T05:28:14.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:14 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:14.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:14 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:14.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:14 smithi005 ceph-mon[29523]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-22T05:28:14.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:14 smithi005 ceph-mon[29523]: Reconfiguring daemon alertmanager.a on smithi005 2024-03-22T05:28:14.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:14 smithi005 ceph-mon[29523]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-22T05:28:14.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:14 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:14.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:14 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:14.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:14 smithi186 ceph-mon[25720]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-22T05:28:14.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:14 smithi186 ceph-mon[25720]: Reconfiguring daemon alertmanager.a on smithi005 2024-03-22T05:28:14.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:14 smithi186 ceph-mon[25720]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-22T05:28:15.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:15 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:28:15.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:15 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:28:15.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:15 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:28:16.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:16 smithi186 ceph-mon[25720]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-22T05:28:16.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:16 smithi005 ceph-mon[29523]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-22T05:28:16.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:16 smithi005 ceph-mon[24793]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-22T05:28:17.478 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:17 smithi005 ceph-mon[24793]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:28:17.478 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:17 smithi005 ceph-mon[29523]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:28:17.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:17 smithi186 ceph-mon[25720]: from='client.24712 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:28:18.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:18 smithi186 ceph-mon[25720]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-22T05:28:18.587 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:18 smithi005 ceph-mon[24793]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-22T05:28:18.587 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:18 smithi005 ceph-mon[29523]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-22T05:28:20.327 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:28:20 smithi005 systemd[1]: Stopping Ceph alertmanager.a for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:28:20.327 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:28:20 smithi005 bash[58876]: Error: no container with name or ID "ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager.a" found: no such container 2024-03-22T05:28:20.620 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:28:20 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[49335]: level=info ts=2024-03-22T05:28:20.327Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-03-22T05:28:20.620 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:28:20 smithi005 podman[58889]: 2024-03-22 05:28:20.348224404 +0000 UTC m=+0.123096961 container died ee43e9b1c4278b9c5af3ae18dd8fc957a70592015277dee687bc39fbe5801362 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-22T05:28:20.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:20 smithi005 ceph-mon[29523]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:28:20.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:20 smithi005 ceph-mon[24793]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:28:20.952 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:28:20 smithi005 podman[58889]: 2024-03-22 05:28:20.711029264 +0000 UTC m=+0.485901821 container cleanup ee43e9b1c4278b9c5af3ae18dd8fc957a70592015277dee687bc39fbe5801362 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-22T05:28:20.953 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:28:20 smithi005 bash[58889]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a 2024-03-22T05:28:20.953 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:28:20 smithi005 bash[58915]: Error: no container with name or ID "ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager.a" found: no such container 2024-03-22T05:28:21.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:20 smithi186 ceph-mon[25720]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:28:21.353 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:28:21 smithi005 podman[58903]: 2024-03-22 05:28:21.020808744 +0000 UTC m=+0.668873283 container remove ee43e9b1c4278b9c5af3ae18dd8fc957a70592015277dee687bc39fbe5801362 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-22T05:28:21.353 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:28:21 smithi005 podman[58903]: 2024-03-22 05:28:21.121445156 +0000 UTC m=+0.769509657 volume remove 175631a1742e331f5ceabda72ba773161ec1b42c76f0650816c7e61895203e63 2024-03-22T05:28:21.353 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:28:21 smithi005 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-22T05:28:21.353 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:28:21 smithi005 systemd[1]: Stopped Ceph alertmanager.a for f59f36e2-e80a-11ee-95cd-87774f69a715. 2024-03-22T05:28:21.353 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:28:21 smithi005 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@alertmanager.a.service: Consumed 1.051s CPU time. 2024-03-22T05:28:21.353 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:28:21 smithi005 systemd[1]: Starting Ceph alertmanager.a for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:28:21.618 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:28:21 smithi005 podman[58993]: 2024-03-22 05:28:21.257523346 +0000 UTC m=+0.019305534 image pull ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4 quay.io/prometheus/alertmanager:v0.23.0 2024-03-22T05:28:21.619 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:28:21 smithi005 podman[58993]: 2024-03-22 05:28:21.355589607 +0000 UTC m=+0.117371793 volume create bc04672c1062bdfcbaacb03f5b5fedd31d84be669ecff6a6528d2edd4d6e5d1b 2024-03-22T05:28:21.619 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:28:21 smithi005 podman[58993]: 2024-03-22 05:28:21.447811927 +0000 UTC m=+0.209594114 container create fd7dc5dbe3e5ea6c4393721c9ae920cc999f81322c3cb7c4627af489527ce955 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-22T05:28:21.952 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:28:21 smithi005 podman[58993]: 2024-03-22 05:28:21.725268071 +0000 UTC m=+0.487050259 container init fd7dc5dbe3e5ea6c4393721c9ae920cc999f81322c3cb7c4627af489527ce955 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-22T05:28:21.952 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:28:21 smithi005 podman[58993]: 2024-03-22 05:28:21.728001884 +0000 UTC m=+0.489784075 container start fd7dc5dbe3e5ea6c4393721c9ae920cc999f81322c3cb7c4627af489527ce955 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-22T05:28:21.953 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:28:21 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[59009]: level=info ts=2024-03-22T05:28:21.738Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-03-22T05:28:21.953 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:28:21 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[59009]: level=info ts=2024-03-22T05:28:21.738Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-03-22T05:28:21.953 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:28:21 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[59009]: level=info ts=2024-03-22T05:28:21.739Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.5 port=9094 2024-03-22T05:28:21.953 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:28:21 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[59009]: level=info ts=2024-03-22T05:28:21.740Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-22T05:28:21.953 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:28:21 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[59009]: level=info ts=2024-03-22T05:28:21.766Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-22T05:28:21.953 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:28:21 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[59009]: level=info ts=2024-03-22T05:28:21.767Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-22T05:28:21.953 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:28:21 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[59009]: level=info ts=2024-03-22T05:28:21.769Z caller=main.go:518 msg=Listening address=:9093 2024-03-22T05:28:21.953 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:28:21 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[59009]: level=info ts=2024-03-22T05:28:21.769Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-03-22T05:28:21.953 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:28:21 smithi005 bash[58993]: fd7dc5dbe3e5ea6c4393721c9ae920cc999f81322c3cb7c4627af489527ce955 2024-03-22T05:28:21.953 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:28:21 smithi005 systemd[1]: Started Ceph alertmanager.a for f59f36e2-e80a-11ee-95cd-87774f69a715. 2024-03-22T05:28:22.702 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:28:22 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:28:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-22T05:28:22.756 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:28:22 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[42595]: ts=2024-03-22T05:28:22.314Z caller=manager.go:609 level=warn component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.5:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.186:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-22T05:28:23.009 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:22 smithi005 ceph-mon[24793]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:28:23.009 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:22 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:23.009 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:22 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:23.009 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:22 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:28:23.009 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:22 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:28:23.009 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:22 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:28:23.009 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:22 smithi005 ceph-mon[29523]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:28:23.009 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:22 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:23.009 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:22 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:23.010 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:22 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:28:23.010 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:22 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:28:23.010 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:22 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:28:23.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:22 smithi186 ceph-mon[25720]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:28:23.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:22 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:23.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:22 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:23.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:22 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:28:23.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:22 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:28:23.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:22 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:28:23.543 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:23 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:28:23] "GET /metrics HTTP/1.1" 200 37878 "" "Prometheus/2.33.4" 2024-03-22T05:28:24.027 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:23 smithi005 ceph-mon[29523]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:28:24.027 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:23 smithi005 ceph-mon[29523]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:28:24.028 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:28:23 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[59009]: level=info ts=2024-03-22T05:28:23.741Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000418291s 2024-03-22T05:28:24.028 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:23 smithi005 ceph-mon[24793]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:28:24.028 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:23 smithi005 ceph-mon[24793]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:28:24.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:23 smithi186 ceph-mon[25720]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:28:24.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:23 smithi186 ceph-mon[25720]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:28:24.876 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:24 smithi005 ceph-mon[29523]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:28:24.876 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:24 smithi005 ceph-mon[24793]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:28:25.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:24 smithi186 ceph-mon[25720]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:28:25.044 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:24 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: 2024-03-22T05:28:24.945+0000 7f0393508700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:28:25.044 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:24 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: Creating ceph-iscsi config... 2024-03-22T05:28:25.044 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:24 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:28:25.044 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:24 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:28:25.044 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:24 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:28:25.044 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:24 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:28:25.044 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:24 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:28:25.044 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:24 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: Traceback (most recent call last): 2024-03-22T05:28:25.044 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:24 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:28:25.044 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:24 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: return _run_code(code, main_globals, None, 2024-03-22T05:28:25.045 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:24 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:28:25.045 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:24 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: exec(code, run_globals) 2024-03-22T05:28:25.045 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:24 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:28:25.045 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:24 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:28:25.045 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:24 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:28:25.045 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:24 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:28:25.045 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:24 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:28:25.046 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:24 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:28:25.046 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:24 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:28:25.046 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:24 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:28:25.047 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:24 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:28:25.047 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:24 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: Traceback (most recent call last): 2024-03-22T05:28:25.047 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:24 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:28:25.047 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:24 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:28:25.047 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:24 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:28:25.047 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:24 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:28:25.047 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:24 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:28:25.047 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:24 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:28:25.048 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:24 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:28:25.048 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:24 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: return future.result(timeout) 2024-03-22T05:28:25.048 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:24 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:28:25.048 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:24 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: return self.__get_result() 2024-03-22T05:28:25.048 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:24 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:28:25.048 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:24 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: raise self._exception 2024-03-22T05:28:25.048 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:24 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:28:25.048 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:24 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: config_blobs=daemon_spec.final_config, 2024-03-22T05:28:25.049 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:24 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:28:25.049 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:24 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:28:25.049 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:24 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:28:25.049 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:24 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: Creating ceph-iscsi config... 2024-03-22T05:28:25.049 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:24 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:28:25.049 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:24 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:28:25.049 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:24 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:28:25.049 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:24 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:28:25.050 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:24 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:28:25.050 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:24 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: Traceback (most recent call last): 2024-03-22T05:28:25.050 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:24 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:28:25.050 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:24 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: return _run_code(code, main_globals, None, 2024-03-22T05:28:25.050 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:24 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:28:25.050 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:24 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: exec(code, run_globals) 2024-03-22T05:28:25.050 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:24 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:28:25.051 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:24 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:28:25.051 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:24 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:28:25.051 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:24 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:28:25.051 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:24 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:28:25.052 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:24 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:28:25.052 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:24 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:28:25.052 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:24 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:28:25.052 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:24 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:28:27.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:26 smithi186 ceph-mon[25720]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:28:27.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:28:27.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:28:27.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:28:27.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:28:27.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:28:27.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:28:27.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:28:27.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:28:27.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:28:27.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:28:27.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:28:27.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:28:27.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:28:27.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:28:27.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:28:27.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:28:27.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:28:27.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:28:27.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:28:27.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:28:27.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:28:27.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:28:27.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:28:27.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:28:27.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:28:27.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:28:27.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:28:27.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:28:27.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: return future.result(timeout) 2024-03-22T05:28:27.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:28:27.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.__get_result() 2024-03-22T05:28:27.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:28:27.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: raise self._exception 2024-03-22T05:28:27.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:28:27.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:28:27.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:28:27.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:28:27.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:28:27.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:28:27.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:28:27.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:28:27.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:28:27.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:28:27.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:28:27.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:28:27.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:28:27.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:28:27.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:28:27.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:28:27.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:28:27.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:28:27.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:28:27.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:28:27.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:28:27.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:28:27.048 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:28:27.048 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:28:27.048 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:28:27.048 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:26 smithi186 ceph-mon[25720]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-22T05:28:27.048 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:26 smithi186 ceph-mon[25720]: Reconfiguring daemon prometheus.a on smithi186 2024-03-22T05:28:27.048 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:26 smithi186 ceph-mon[25720]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:28:27.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:26 smithi005 ceph-mon[29523]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:28:27.202 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:28:27.203 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:28:27.203 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:28:27.203 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:28:27.203 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:28:27.203 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:28:27.203 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:28:27.203 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:28:27.203 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:28:27.203 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:28:27.203 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:28:27.203 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:28:27.203 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:28:27.203 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:28:27.203 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:28:27.204 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:28:27.204 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:28:27.204 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:28:27.204 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:28:27.204 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:28:27.204 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:28:27.204 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:28:27.204 INFO:journalctl@ceph.mon.c.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:28:27.204 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:28:27.204 INFO:journalctl@ceph.mon.c.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:28:27.204 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:28:27.204 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:28:27.204 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:28:27.204 INFO:journalctl@ceph.mon.c.smithi005.stdout: return future.result(timeout) 2024-03-22T05:28:27.204 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:28:27.205 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.__get_result() 2024-03-22T05:28:27.205 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:28:27.205 INFO:journalctl@ceph.mon.c.smithi005.stdout: raise self._exception 2024-03-22T05:28:27.205 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:28:27.205 INFO:journalctl@ceph.mon.c.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:28:27.205 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:28:27.205 INFO:journalctl@ceph.mon.c.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:28:27.205 INFO:journalctl@ceph.mon.c.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:28:27.205 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:28:27.205 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:28:27.205 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:28:27.205 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:28:27.205 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:28:27.205 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:28:27.205 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:28:27.205 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:28:27.206 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:28:27.206 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:28:27.206 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:28:27.206 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:28:27.206 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:28:27.206 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:28:27.206 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:28:27.206 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:26 smithi005 ceph-mon[24793]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:28:27.206 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:28:27.206 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:28:27.207 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:28:27.207 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:28:27.207 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:28:27.207 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:28:27.207 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:28:27.207 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:28:27.207 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:28:27.207 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:28:27.207 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:28:27.207 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:28:27.207 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:28:27.207 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:28:27.207 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:28:27.207 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:28:27.208 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:28:27.208 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:28:27.208 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:28:27.208 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:28:27.208 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:28:27.208 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:28:27.208 INFO:journalctl@ceph.mon.a.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:28:27.208 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:28:27.208 INFO:journalctl@ceph.mon.a.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:28:27.208 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:28:27.208 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:28:27.208 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:28:27.209 INFO:journalctl@ceph.mon.a.smithi005.stdout: return future.result(timeout) 2024-03-22T05:28:27.209 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:28:27.209 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.__get_result() 2024-03-22T05:28:27.209 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:28:27.209 INFO:journalctl@ceph.mon.a.smithi005.stdout: raise self._exception 2024-03-22T05:28:27.209 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:28:27.209 INFO:journalctl@ceph.mon.a.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:28:27.209 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:28:27.209 INFO:journalctl@ceph.mon.a.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:28:27.209 INFO:journalctl@ceph.mon.a.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:28:27.209 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:28:27.209 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:28:27.209 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:28:27.209 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:28:27.209 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:28:27.210 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:28:27.210 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:28:27.210 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:28:27.210 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:28:27.210 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:28:27.210 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:28:27.210 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:28:27.210 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:28:27.210 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:28:27.210 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:28:27.210 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:28:27.210 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:28:27.210 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:28:27.211 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:28:27.211 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:28:27.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:26 smithi005 ceph-mon[24793]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-22T05:28:27.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:26 smithi005 ceph-mon[24793]: Reconfiguring daemon prometheus.a on smithi186 2024-03-22T05:28:27.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:26 smithi005 ceph-mon[24793]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:28:27.211 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:28:27.211 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:28:27.211 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:28:27.211 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:28:27.211 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:28:27.211 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:26 smithi005 ceph-mon[29523]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-22T05:28:27.212 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:26 smithi005 ceph-mon[29523]: Reconfiguring daemon prometheus.a on smithi186 2024-03-22T05:28:27.212 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:26 smithi005 ceph-mon[29523]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:28:29.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:28 smithi186 ceph-mon[25720]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:28:29.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:28 smithi005 ceph-mon[24793]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:28:29.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:28 smithi005 ceph-mon[29523]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:28:31.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:30 smithi186 ceph-mon[25720]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-22T05:28:31.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:30 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:28:31.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:30 smithi005 ceph-mon[24793]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-22T05:28:31.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:30 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:28:31.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:30 smithi005 ceph-mon[29523]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-22T05:28:31.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:30 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:28:32.202 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:28:31 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[59009]: level=info ts=2024-03-22T05:28:31.744Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.003667625s 2024-03-22T05:28:32.702 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:28:32 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[25021]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:28:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-22T05:28:32.767 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:28:32 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[42595]: ts=2024-03-22T05:28:32.313Z caller=manager.go:609 level=warn component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.5:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.186:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-22T05:28:33.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:32 smithi186 ceph-mon[25720]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-22T05:28:33.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:32 smithi005 ceph-mon[29523]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-22T05:28:33.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:32 smithi005 ceph-mon[24793]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-22T05:28:33.543 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:33 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:28:33] "GET /metrics HTTP/1.1" 200 37885 "" "Prometheus/2.33.4" 2024-03-22T05:28:34.493 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:34 smithi186 ceph-mon[25720]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-22T05:28:34.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:34 smithi005 ceph-mon[24793]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-22T05:28:34.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:34 smithi005 ceph-mon[29523]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-22T05:28:36.180 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:28:36 smithi186 systemd[1]: Stopping Ceph prometheus.a for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:28:36.181 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:28:36 smithi186 bash[49100]: Error: no container with name or ID "ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus.a" found: no such container 2024-03-22T05:28:36.465 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:28:36 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[42595]: ts=2024-03-22T05:28:36.181Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-22T05:28:36.466 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:28:36 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[42595]: ts=2024-03-22T05:28:36.181Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-03-22T05:28:36.466 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:28:36 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[42595]: ts=2024-03-22T05:28:36.181Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-03-22T05:28:36.466 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:28:36 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[42595]: ts=2024-03-22T05:28:36.181Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-03-22T05:28:36.466 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:28:36 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[42595]: ts=2024-03-22T05:28:36.181Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-03-22T05:28:36.466 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:28:36 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[42595]: ts=2024-03-22T05:28:36.181Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-03-22T05:28:36.466 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:28:36 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[42595]: ts=2024-03-22T05:28:36.182Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-22T05:28:36.466 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:28:36 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[42595]: ts=2024-03-22T05:28:36.182Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-03-22T05:28:36.466 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:28:36 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[42595]: ts=2024-03-22T05:28:36.182Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-03-22T05:28:36.466 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:28:36 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[42595]: ts=2024-03-22T05:28:36.182Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-03-22T05:28:36.466 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:28:36 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[42595]: ts=2024-03-22T05:28:36.182Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-03-22T05:28:36.466 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:28:36 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[42595]: ts=2024-03-22T05:28:36.182Z caller=main.go:1066 level=info msg="See you next time!" 2024-03-22T05:28:36.467 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:28:36 smithi186 podman[49113]: 2024-03-22 05:28:36.191798272 +0000 UTC m=+0.137945548 container died f520626945ac93d954db1d7a0d807945f01fbffaf6947b3a66590fcc2f0ee9e6 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-22T05:28:36.758 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:28:36 smithi186 podman[49113]: 2024-03-22 05:28:36.581577642 +0000 UTC m=+0.527724911 container cleanup f520626945ac93d954db1d7a0d807945f01fbffaf6947b3a66590fcc2f0ee9e6 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-22T05:28:36.758 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:28:36 smithi186 bash[49113]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a 2024-03-22T05:28:36.759 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:28:36 smithi186 bash[49139]: Error: no container with name or ID "ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus.a" found: no such container 2024-03-22T05:28:36.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:36 smithi005 ceph-mon[24793]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-22T05:28:36.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:36 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/1594411620' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-22T05:28:36.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:36 smithi005 ceph-mon[29523]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-22T05:28:36.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:36 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/1594411620' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-22T05:28:37.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:36 smithi186 ceph-mon[25720]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-22T05:28:37.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:36 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/1594411620' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-22T05:28:37.044 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:28:36 smithi186 podman[49127]: 2024-03-22 05:28:36.849785511 +0000 UTC m=+0.659997057 container remove f520626945ac93d954db1d7a0d807945f01fbffaf6947b3a66590fcc2f0ee9e6 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-22T05:28:37.044 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:28:36 smithi186 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-03-22T05:28:37.044 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:28:36 smithi186 systemd[1]: Stopped Ceph prometheus.a for f59f36e2-e80a-11ee-95cd-87774f69a715. 2024-03-22T05:28:37.044 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:28:36 smithi186 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@prometheus.a.service: Consumed 1.983s CPU time. 2024-03-22T05:28:37.044 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:28:36 smithi186 systemd[1]: Starting Ceph prometheus.a for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:28:37.337 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:28:37 smithi186 podman[49210]: 2024-03-22 05:28:36.988058552 +0000 UTC m=+0.019489785 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-03-22T05:28:37.337 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:28:37 smithi186 podman[49210]: 2024-03-22 05:28:37.125864829 +0000 UTC m=+0.157296062 container create c3a69c1f7e79f68a37cb6b6c272106ec38f748a73e302b0caec7f651c4e5ad09 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-22T05:28:37.338 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:28:37 smithi186 podman[49210]: 2024-03-22 05:28:37.311345176 +0000 UTC m=+0.342776409 container init c3a69c1f7e79f68a37cb6b6c272106ec38f748a73e302b0caec7f651c4e5ad09 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-22T05:28:37.338 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:28:37 smithi186 podman[49210]: 2024-03-22 05:28:37.314055983 +0000 UTC m=+0.345487229 container start c3a69c1f7e79f68a37cb6b6c272106ec38f748a73e302b0caec7f651c4e5ad09 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-22T05:28:37.759 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:28:37 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:28:37.338Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-03-22T05:28:37.759 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:28:37 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:28:37.338Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-03-22T05:28:37.759 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:28:37 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:28:37.338Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-03-22T05:28:37.759 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:28:37 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:28:37.338Z caller=main.go:518 level=info host_details="(Linux 5.14.0-430.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024 x86_64 smithi186 (none))" 2024-03-22T05:28:37.760 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:28:37 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:28:37.338Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-22T05:28:37.760 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:28:37 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:28:37.338Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-22T05:28:37.760 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:28:37 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:28:37.340Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-03-22T05:28:37.760 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:28:37 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:28:37.340Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-03-22T05:28:37.760 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:28:37 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:28:37.341Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-03-22T05:28:37.760 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:28:37 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:28:37.342Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-22T05:28:37.760 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:28:37 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:28:37.342Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.457µs 2024-03-22T05:28:37.760 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:28:37 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:28:37.342Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-22T05:28:37.760 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:28:37 smithi186 bash[49210]: c3a69c1f7e79f68a37cb6b6c272106ec38f748a73e302b0caec7f651c4e5ad09 2024-03-22T05:28:37.760 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:28:37 smithi186 systemd[1]: Started Ceph prometheus.a for f59f36e2-e80a-11ee-95cd-87774f69a715. 2024-03-22T05:28:37.760 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:28:37 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:28:37.554Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=2 2024-03-22T05:28:38.028 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:28:38 smithi186 systemd[1]: Stopping Ceph node-exporter.b for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:28:38.028 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:37 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/3764182892' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/2830010477"}]: dispatch 2024-03-22T05:28:38.028 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:37 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:38.028 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:37 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:38.028 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:28:37 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:28:37.791Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=2 2024-03-22T05:28:38.029 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:28:37 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:28:37.791Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=2 2024-03-22T05:28:38.029 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:28:37 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:28:37.791Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=26.868µs wal_replay_duration=448.891091ms total_replay_duration=448.930401ms 2024-03-22T05:28:38.029 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:28:37 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:28:37.794Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-03-22T05:28:38.029 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:28:37 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:28:37.794Z caller=main.go:947 level=info msg="TSDB started" 2024-03-22T05:28:38.029 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:28:37 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:28:37.794Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-22T05:28:38.029 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:28:37 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:28:37.816Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=21.921849ms db_storage=768ns remote_storage=1.344µs web_handler=336ns query_engine=665ns scrape=378.519µs scrape_sd=93.935µs notify=15.728µs notify_sd=22.417µs rules=21.090214ms 2024-03-22T05:28:38.029 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:28:37 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:28:37.816Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-03-22T05:28:38.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:37 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/3764182892' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/2830010477"}]: dispatch 2024-03-22T05:28:38.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:37 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:38.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:37 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:38.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:37 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/3764182892' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/2830010477"}]: dispatch 2024-03-22T05:28:38.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:37 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:38.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:37 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:38.293 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:28:38 smithi186 bash[49334]: Error: no container with name or ID "ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter.b" found: no such container 2024-03-22T05:28:38.293 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:28:38 smithi186 podman[49345]: 2024-03-22 05:28:38.166885463 +0000 UTC m=+0.129827445 container died e1c552dad9306f79982a50546495ed3b583865bb1383eee35b99463ed491349d (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-22T05:28:38.757 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:28:38 smithi186 podman[49345]: 2024-03-22 05:28:38.53971097 +0000 UTC m=+0.502652959 container cleanup e1c552dad9306f79982a50546495ed3b583865bb1383eee35b99463ed491349d (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-22T05:28:38.757 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:28:38 smithi186 bash[49345]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b 2024-03-22T05:28:39.038 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:38 smithi186 ceph-mon[25720]: Reconfiguring node-exporter.b (dependencies changed)... 2024-03-22T05:28:39.038 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:38 smithi186 ceph-mon[25720]: Reconfiguring daemon node-exporter.b on smithi186 2024-03-22T05:28:39.039 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:38 smithi186 ceph-mon[25720]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-22T05:28:39.039 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:38 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/3764182892' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/2830010477"}]': finished 2024-03-22T05:28:39.039 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:38 smithi186 ceph-mon[25720]: osdmap e95: 8 total, 8 up, 8 in 2024-03-22T05:28:39.039 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:38 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/59972611' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/2754168109"}]: dispatch 2024-03-22T05:28:39.039 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:38 smithi186 ceph-mon[25720]: from='client.? ' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/2754168109"}]: dispatch 2024-03-22T05:28:39.040 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:28:38 smithi186 podman[49359]: 2024-03-22 05:28:38.757969636 +0000 UTC m=+0.595947161 container remove e1c552dad9306f79982a50546495ed3b583865bb1383eee35b99463ed491349d (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-22T05:28:39.040 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:28:38 smithi186 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-03-22T05:28:39.040 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:28:38 smithi186 bash[49374]: Error: no container with name or ID "ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter.b" found: no such container 2024-03-22T05:28:39.040 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:28:38 smithi186 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-03-22T05:28:39.041 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:28:38 smithi186 systemd[1]: Stopped Ceph node-exporter.b for f59f36e2-e80a-11ee-95cd-87774f69a715. 2024-03-22T05:28:39.041 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:28:38 smithi186 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@node-exporter.b.service: Consumed 3.795s CPU time. 2024-03-22T05:28:39.041 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:28:38 smithi186 systemd[1]: Starting Ceph node-exporter.b for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:28:39.103 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:38 smithi005 ceph-mon[24793]: Reconfiguring node-exporter.b (dependencies changed)... 2024-03-22T05:28:39.103 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:38 smithi005 ceph-mon[24793]: Reconfiguring daemon node-exporter.b on smithi186 2024-03-22T05:28:39.103 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:38 smithi005 ceph-mon[24793]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-22T05:28:39.104 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:38 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/3764182892' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/2830010477"}]': finished 2024-03-22T05:28:39.104 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:38 smithi005 ceph-mon[24793]: osdmap e95: 8 total, 8 up, 8 in 2024-03-22T05:28:39.104 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:38 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/59972611' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/2754168109"}]: dispatch 2024-03-22T05:28:39.104 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:38 smithi005 ceph-mon[24793]: from='client.? ' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/2754168109"}]: dispatch 2024-03-22T05:28:39.104 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:38 smithi005 ceph-mon[29523]: Reconfiguring node-exporter.b (dependencies changed)... 2024-03-22T05:28:39.104 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:38 smithi005 ceph-mon[29523]: Reconfiguring daemon node-exporter.b on smithi186 2024-03-22T05:28:39.104 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:38 smithi005 ceph-mon[29523]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-22T05:28:39.104 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:38 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/3764182892' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/2830010477"}]': finished 2024-03-22T05:28:39.104 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:38 smithi005 ceph-mon[29523]: osdmap e95: 8 total, 8 up, 8 in 2024-03-22T05:28:39.104 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:38 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/59972611' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/2754168109"}]: dispatch 2024-03-22T05:28:39.105 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:38 smithi005 ceph-mon[29523]: from='client.? ' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/2754168109"}]: dispatch 2024-03-22T05:28:39.294 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:28:39 smithi186 podman[49451]: 2024-03-22 05:28:38.940937961 +0000 UTC m=+0.017371799 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-03-22T05:28:39.294 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:28:39 smithi186 podman[49451]: 2024-03-22 05:28:39.075778706 +0000 UTC m=+0.152212541 container create 74aa1181ff717bd9b36ec7e639ed394fff71031a1d4ffc165c74454aad00f662 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-22T05:28:39.294 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:28:39 smithi186 podman[49451]: 2024-03-22 05:28:39.278265341 +0000 UTC m=+0.354699174 container init 74aa1181ff717bd9b36ec7e639ed394fff71031a1d4ffc165c74454aad00f662 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-22T05:28:39.294 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:28:39 smithi186 podman[49451]: 2024-03-22 05:28:39.280963182 +0000 UTC m=+0.357397021 container start 74aa1181ff717bd9b36ec7e639ed394fff71031a1d4ffc165c74454aad00f662 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-22T05:28:39.294 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:28:39 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[49467]: ts=2024-03-22T05:28:39.284Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-03-22T05:28:39.294 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:28:39 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[49467]: ts=2024-03-22T05:28:39.284Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-03-22T05:28:39.294 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:28:39 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[49467]: ts=2024-03-22T05:28:39.284Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-03-22T05:28:39.294 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:28:39 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[49467]: ts=2024-03-22T05:28:39.284Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-22T05:28:39.294 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:28:39 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[49467]: ts=2024-03-22T05:28:39.285Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-03-22T05:28:39.295 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:28:39 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[49467]: ts=2024-03-22T05:28:39.285Z caller=node_exporter.go:115 level=info collector=arp 2024-03-22T05:28:39.295 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:28:39 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[49467]: ts=2024-03-22T05:28:39.285Z caller=node_exporter.go:115 level=info collector=bcache 2024-03-22T05:28:39.295 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:28:39 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[49467]: ts=2024-03-22T05:28:39.285Z caller=node_exporter.go:115 level=info collector=bonding 2024-03-22T05:28:39.295 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:28:39 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[49467]: ts=2024-03-22T05:28:39.285Z caller=node_exporter.go:115 level=info collector=btrfs 2024-03-22T05:28:39.295 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:28:39 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[49467]: ts=2024-03-22T05:28:39.285Z caller=node_exporter.go:115 level=info collector=conntrack 2024-03-22T05:28:39.295 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:28:39 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[49467]: ts=2024-03-22T05:28:39.285Z caller=node_exporter.go:115 level=info collector=cpu 2024-03-22T05:28:39.295 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:28:39 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[49467]: ts=2024-03-22T05:28:39.285Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-03-22T05:28:39.295 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:28:39 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[49467]: ts=2024-03-22T05:28:39.285Z caller=node_exporter.go:115 level=info collector=diskstats 2024-03-22T05:28:39.296 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:28:39 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[49467]: ts=2024-03-22T05:28:39.285Z caller=node_exporter.go:115 level=info collector=dmi 2024-03-22T05:28:39.296 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:28:39 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[49467]: ts=2024-03-22T05:28:39.285Z caller=node_exporter.go:115 level=info collector=edac 2024-03-22T05:28:39.296 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:28:39 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[49467]: ts=2024-03-22T05:28:39.285Z caller=node_exporter.go:115 level=info collector=entropy 2024-03-22T05:28:39.296 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:28:39 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[49467]: ts=2024-03-22T05:28:39.285Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-03-22T05:28:39.296 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:28:39 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[49467]: ts=2024-03-22T05:28:39.285Z caller=node_exporter.go:115 level=info collector=filefd 2024-03-22T05:28:39.296 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:28:39 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[49467]: ts=2024-03-22T05:28:39.285Z caller=node_exporter.go:115 level=info collector=filesystem 2024-03-22T05:28:39.296 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:28:39 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[49467]: ts=2024-03-22T05:28:39.285Z caller=node_exporter.go:115 level=info collector=hwmon 2024-03-22T05:28:39.296 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:28:39 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[49467]: ts=2024-03-22T05:28:39.285Z caller=node_exporter.go:115 level=info collector=infiniband 2024-03-22T05:28:39.296 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:28:39 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[49467]: ts=2024-03-22T05:28:39.285Z caller=node_exporter.go:115 level=info collector=ipvs 2024-03-22T05:28:39.296 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:28:39 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[49467]: ts=2024-03-22T05:28:39.285Z caller=node_exporter.go:115 level=info collector=loadavg 2024-03-22T05:28:39.296 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:28:39 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[49467]: ts=2024-03-22T05:28:39.285Z caller=node_exporter.go:115 level=info collector=mdadm 2024-03-22T05:28:39.296 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:28:39 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[49467]: ts=2024-03-22T05:28:39.285Z caller=node_exporter.go:115 level=info collector=meminfo 2024-03-22T05:28:39.296 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:28:39 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[49467]: ts=2024-03-22T05:28:39.285Z caller=node_exporter.go:115 level=info collector=netclass 2024-03-22T05:28:39.296 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:28:39 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[49467]: ts=2024-03-22T05:28:39.285Z caller=node_exporter.go:115 level=info collector=netdev 2024-03-22T05:28:39.297 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:28:39 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[49467]: ts=2024-03-22T05:28:39.285Z caller=node_exporter.go:115 level=info collector=netstat 2024-03-22T05:28:39.297 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:28:39 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[49467]: ts=2024-03-22T05:28:39.285Z caller=node_exporter.go:115 level=info collector=nfs 2024-03-22T05:28:39.297 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:28:39 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[49467]: ts=2024-03-22T05:28:39.285Z caller=node_exporter.go:115 level=info collector=nfsd 2024-03-22T05:28:39.297 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:28:39 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[49467]: ts=2024-03-22T05:28:39.285Z caller=node_exporter.go:115 level=info collector=nvme 2024-03-22T05:28:39.297 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:28:39 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[49467]: ts=2024-03-22T05:28:39.285Z caller=node_exporter.go:115 level=info collector=os 2024-03-22T05:28:39.297 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:28:39 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[49467]: ts=2024-03-22T05:28:39.285Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-03-22T05:28:39.297 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:28:39 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[49467]: ts=2024-03-22T05:28:39.285Z caller=node_exporter.go:115 level=info collector=pressure 2024-03-22T05:28:39.297 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:28:39 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[49467]: ts=2024-03-22T05:28:39.285Z caller=node_exporter.go:115 level=info collector=rapl 2024-03-22T05:28:39.297 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:28:39 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[49467]: ts=2024-03-22T05:28:39.285Z caller=node_exporter.go:115 level=info collector=schedstat 2024-03-22T05:28:39.297 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:28:39 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[49467]: ts=2024-03-22T05:28:39.285Z caller=node_exporter.go:115 level=info collector=sockstat 2024-03-22T05:28:39.297 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:28:39 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[49467]: ts=2024-03-22T05:28:39.285Z caller=node_exporter.go:115 level=info collector=softnet 2024-03-22T05:28:39.297 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:28:39 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[49467]: ts=2024-03-22T05:28:39.285Z caller=node_exporter.go:115 level=info collector=stat 2024-03-22T05:28:39.297 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:28:39 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[49467]: ts=2024-03-22T05:28:39.285Z caller=node_exporter.go:115 level=info collector=tapestats 2024-03-22T05:28:39.297 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:28:39 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[49467]: ts=2024-03-22T05:28:39.285Z caller=node_exporter.go:115 level=info collector=textfile 2024-03-22T05:28:39.297 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:28:39 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[49467]: ts=2024-03-22T05:28:39.285Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-03-22T05:28:39.298 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:28:39 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[49467]: ts=2024-03-22T05:28:39.285Z caller=node_exporter.go:115 level=info collector=time 2024-03-22T05:28:39.298 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:28:39 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[49467]: ts=2024-03-22T05:28:39.285Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-03-22T05:28:39.298 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:28:39 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[49467]: ts=2024-03-22T05:28:39.285Z caller=node_exporter.go:115 level=info collector=uname 2024-03-22T05:28:39.298 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:28:39 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[49467]: ts=2024-03-22T05:28:39.285Z caller=node_exporter.go:115 level=info collector=vmstat 2024-03-22T05:28:39.298 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:28:39 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[49467]: ts=2024-03-22T05:28:39.285Z caller=node_exporter.go:115 level=info collector=xfs 2024-03-22T05:28:39.298 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:28:39 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[49467]: ts=2024-03-22T05:28:39.285Z caller=node_exporter.go:115 level=info collector=zfs 2024-03-22T05:28:39.298 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:28:39 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[49467]: ts=2024-03-22T05:28:39.285Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-03-22T05:28:39.298 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:28:39 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[49467]: ts=2024-03-22T05:28:39.285Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-03-22T05:28:39.780 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:28:39 smithi186 bash[49451]: 74aa1181ff717bd9b36ec7e639ed394fff71031a1d4ffc165c74454aad00f662 2024-03-22T05:28:39.780 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:28:39 smithi186 systemd[1]: Started Ceph node-exporter.b for f59f36e2-e80a-11ee-95cd-87774f69a715. 2024-03-22T05:28:40.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:39 smithi186 ceph-mon[25720]: from='client.? ' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/2754168109"}]': finished 2024-03-22T05:28:40.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:39 smithi186 ceph-mon[25720]: osdmap e96: 8 total, 8 up, 8 in 2024-03-22T05:28:40.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:39 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/634509531' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6800/3281603100"}]: dispatch 2024-03-22T05:28:40.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:39 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:40.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:39 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:40.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:39 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:40.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:39 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:40.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:39 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-22T05:28:40.113 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:39 smithi005 ceph-mon[24793]: from='client.? ' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/2754168109"}]': finished 2024-03-22T05:28:40.113 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:39 smithi005 ceph-mon[24793]: osdmap e96: 8 total, 8 up, 8 in 2024-03-22T05:28:40.113 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:39 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/634509531' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6800/3281603100"}]: dispatch 2024-03-22T05:28:40.113 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:39 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:40.113 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:39 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:40.113 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:39 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:40.113 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:39 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:40.114 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:39 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-22T05:28:40.114 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:39 smithi005 ceph-mon[29523]: from='client.? ' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/2754168109"}]': finished 2024-03-22T05:28:40.114 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:39 smithi005 ceph-mon[29523]: osdmap e96: 8 total, 8 up, 8 in 2024-03-22T05:28:40.114 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:39 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/634509531' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6800/3281603100"}]: dispatch 2024-03-22T05:28:40.114 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:39 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:40.114 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:39 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:40.114 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:39 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:40.114 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:39 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:40.114 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:39 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-22T05:28:41.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:40 smithi186 ceph-mon[25720]: Reconfiguring grafana.a (dependencies changed)... 2024-03-22T05:28:41.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:40 smithi186 ceph-mon[25720]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-22T05:28:41.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:40 smithi186 ceph-mon[25720]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-22T05:28:41.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:40 smithi186 ceph-mon[25720]: Reconfiguring daemon grafana.a on smithi186 2024-03-22T05:28:41.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:40 smithi186 ceph-mon[25720]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:28:41.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:40 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/634509531' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6800/3281603100"}]': finished 2024-03-22T05:28:41.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:40 smithi186 ceph-mon[25720]: osdmap e97: 8 total, 8 up, 8 in 2024-03-22T05:28:41.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:40 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/674697316' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/4022295834"}]: dispatch 2024-03-22T05:28:41.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:40 smithi005 ceph-mon[24793]: Reconfiguring grafana.a (dependencies changed)... 2024-03-22T05:28:41.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:40 smithi005 ceph-mon[24793]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-22T05:28:41.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:40 smithi005 ceph-mon[24793]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-22T05:28:41.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:40 smithi005 ceph-mon[24793]: Reconfiguring daemon grafana.a on smithi186 2024-03-22T05:28:41.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:40 smithi005 ceph-mon[24793]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:28:41.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:40 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/634509531' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6800/3281603100"}]': finished 2024-03-22T05:28:41.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:40 smithi005 ceph-mon[24793]: osdmap e97: 8 total, 8 up, 8 in 2024-03-22T05:28:41.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:40 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/674697316' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/4022295834"}]: dispatch 2024-03-22T05:28:41.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:40 smithi005 ceph-mon[29523]: Reconfiguring grafana.a (dependencies changed)... 2024-03-22T05:28:41.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:40 smithi005 ceph-mon[29523]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-22T05:28:41.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:40 smithi005 ceph-mon[29523]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-22T05:28:41.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:40 smithi005 ceph-mon[29523]: Reconfiguring daemon grafana.a on smithi186 2024-03-22T05:28:41.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:40 smithi005 ceph-mon[29523]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:28:41.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:40 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/634509531' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6800/3281603100"}]': finished 2024-03-22T05:28:41.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:40 smithi005 ceph-mon[29523]: osdmap e97: 8 total, 8 up, 8 in 2024-03-22T05:28:41.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:40 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/674697316' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/4022295834"}]: dispatch 2024-03-22T05:28:42.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:41 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/674697316' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/4022295834"}]': finished 2024-03-22T05:28:42.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:41 smithi005 ceph-mon[24793]: osdmap e98: 8 total, 8 up, 8 in 2024-03-22T05:28:42.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:41 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/2766629896' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6801/3281603100"}]: dispatch 2024-03-22T05:28:42.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:41 smithi005 ceph-mon[24793]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:28:42.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:41 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/674697316' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/4022295834"}]': finished 2024-03-22T05:28:42.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:41 smithi005 ceph-mon[29523]: osdmap e98: 8 total, 8 up, 8 in 2024-03-22T05:28:42.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:41 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/2766629896' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6801/3281603100"}]: dispatch 2024-03-22T05:28:42.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:41 smithi005 ceph-mon[29523]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:28:42.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:41 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/674697316' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/4022295834"}]': finished 2024-03-22T05:28:42.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:41 smithi186 ceph-mon[25720]: osdmap e98: 8 total, 8 up, 8 in 2024-03-22T05:28:42.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:41 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/2766629896' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6801/3281603100"}]: dispatch 2024-03-22T05:28:42.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:41 smithi186 ceph-mon[25720]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:28:43.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:42 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/2766629896' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6801/3281603100"}]': finished 2024-03-22T05:28:43.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:42 smithi005 ceph-mon[24793]: osdmap e99: 8 total, 8 up, 8 in 2024-03-22T05:28:43.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:42 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/873616983' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/2894494732"}]: dispatch 2024-03-22T05:28:43.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:42 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/2766629896' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6801/3281603100"}]': finished 2024-03-22T05:28:43.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:42 smithi005 ceph-mon[29523]: osdmap e99: 8 total, 8 up, 8 in 2024-03-22T05:28:43.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:42 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/873616983' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/2894494732"}]: dispatch 2024-03-22T05:28:43.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:42 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/2766629896' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6801/3281603100"}]': finished 2024-03-22T05:28:43.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:42 smithi186 ceph-mon[25720]: osdmap e99: 8 total, 8 up, 8 in 2024-03-22T05:28:43.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:42 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/873616983' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/2894494732"}]: dispatch 2024-03-22T05:28:43.794 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:28:43 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:28:43.503Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi186\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi186\", job=\"node\", machine=\"x86_64\", nodename=\"smithi186\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"f59f36e2-e80a-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi186\", job=\"node\", machine=\"x86_64\", nodename=\"smithi186\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-22T05:28:44.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:43 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/873616983' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/2894494732"}]': finished 2024-03-22T05:28:44.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:43 smithi005 ceph-mon[24793]: osdmap e100: 8 total, 8 up, 8 in 2024-03-22T05:28:44.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:43 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/2514920942' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/3115554819"}]: dispatch 2024-03-22T05:28:44.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:43 smithi005 ceph-mon[24793]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:28:44.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:43 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/873616983' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/2894494732"}]': finished 2024-03-22T05:28:44.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:43 smithi005 ceph-mon[29523]: osdmap e100: 8 total, 8 up, 8 in 2024-03-22T05:28:44.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:43 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/2514920942' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/3115554819"}]: dispatch 2024-03-22T05:28:44.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:43 smithi005 ceph-mon[29523]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:28:44.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:43 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/873616983' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/2894494732"}]': finished 2024-03-22T05:28:44.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:43 smithi186 ceph-mon[25720]: osdmap e100: 8 total, 8 up, 8 in 2024-03-22T05:28:44.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:43 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/2514920942' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/3115554819"}]: dispatch 2024-03-22T05:28:44.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:43 smithi186 ceph-mon[25720]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:28:45.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:44 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/2514920942' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/3115554819"}]': finished 2024-03-22T05:28:45.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:44 smithi005 ceph-mon[24793]: osdmap e101: 8 total, 8 up, 8 in 2024-03-22T05:28:45.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:44 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:28:45.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:44 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/2514920942' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/3115554819"}]': finished 2024-03-22T05:28:45.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:44 smithi005 ceph-mon[29523]: osdmap e101: 8 total, 8 up, 8 in 2024-03-22T05:28:45.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:44 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:28:45.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:44 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/2514920942' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/3115554819"}]': finished 2024-03-22T05:28:45.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:44 smithi186 ceph-mon[25720]: osdmap e101: 8 total, 8 up, 8 in 2024-03-22T05:28:45.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:44 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:28:46.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:45 smithi005 ceph-mon[29523]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:28:46.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:45 smithi005 ceph-mon[24793]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:28:46.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:45 smithi186 ceph-mon[25720]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:28:47.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:46 smithi005 ceph-mon[24793]: from='client.24991 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:28:47.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:46 smithi005 ceph-mon[29523]: from='client.24991 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:28:47.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:46 smithi186 ceph-mon[25720]: from='client.24991 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:28:48.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:47 smithi005 ceph-mon[24793]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:28:48.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:47 smithi005 ceph-mon[29523]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:28:48.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:47 smithi186 ceph-mon[25720]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:28:50.856 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:50 smithi186 ceph-mon[25720]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 647 B/s rd, 0 op/s 2024-03-22T05:28:51.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:50 smithi005 ceph-mon[24793]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 647 B/s rd, 0 op/s 2024-03-22T05:28:51.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:50 smithi005 ceph-mon[29523]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 647 B/s rd, 0 op/s 2024-03-22T05:28:51.294 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:28:50 smithi186 systemd[1]: Stopping Ceph grafana.a for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:28:51.294 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:28:50 smithi186 bash[49755]: Error: no container with name or ID "ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana.a" found: no such container 2024-03-22T05:28:51.294 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:28:51 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[41790]: t=2024-03-22T05:28:51+0000 lvl=info msg="Shutdown started" logger=server reason="System signal: terminated" 2024-03-22T05:28:51.294 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:28:51 smithi186 podman[49769]: 2024-03-22 05:28:51.078676861 +0000 UTC m=+0.132561177 container died 27324e98a8bba35073c6b755d0d7719ad44aecb9c43c027b09f3ba3cd51f4edf (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a, distribution-scope=public, release=236.1648460182, summary=Grafana Container configured for Ceph mgr/dashboard integration, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, maintainer=Paul Cuzner , build-date=2022-03-28T10:36:18.413762, com.redhat.component=ubi8-container, io.buildah.version=1.24.2, vendor=Red Hat, Inc., io.openshift.expose-services=, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.tags=base rhel8, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8.5, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, description=Ceph Grafana Container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vcs-type=git, name=ubi8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-03-22T05:28:51.702 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:51 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:28:51] "GET /metrics HTTP/1.1" 200 37885 "" "Prometheus/2.33.4" 2024-03-22T05:28:51.702 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:28:51 smithi186 podman[49769]: 2024-03-22 05:28:51.501864121 +0000 UTC m=+0.555748438 container cleanup 27324e98a8bba35073c6b755d0d7719ad44aecb9c43c027b09f3ba3cd51f4edf (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a, distribution-scope=public, release=236.1648460182, version=8.5, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.tags=base rhel8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, description=Ceph Grafana Container, maintainer=Paul Cuzner , name=ubi8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=ubi8-container, vendor=Red Hat, Inc., summary=Grafana Container configured for Ceph mgr/dashboard integration, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vcs-type=git, io.openshift.expose-services=, io.buildah.version=1.24.2, architecture=x86_64, build-date=2022-03-28T10:36:18.413762) 2024-03-22T05:28:51.702 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:28:51 smithi186 bash[49769]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a 2024-03-22T05:28:51.702 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:28:51 smithi186 bash[49796]: Error: no container with name or ID "ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana.a" found: no such container 2024-03-22T05:28:51.961 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:28:51 smithi186 podman[49783]: 2024-03-22 05:28:51.702962279 +0000 UTC m=+0.624811043 container remove 27324e98a8bba35073c6b755d0d7719ad44aecb9c43c027b09f3ba3cd51f4edf (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a, io.openshift.tags=base rhel8, description=Ceph Grafana Container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-type=git, build-date=2022-03-28T10:36:18.413762, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vendor=Red Hat, Inc., version=8.5, maintainer=Paul Cuzner , architecture=x86_64, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.buildah.version=1.24.2, com.redhat.component=ubi8-container, distribution-scope=public, release=236.1648460182, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=Red Hat Universal Base Image 8, name=ubi8) 2024-03-22T05:28:51.961 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:28:51 smithi186 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@grafana.a.service: Deactivated successfully. 2024-03-22T05:28:51.962 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:28:51 smithi186 systemd[1]: Stopped Ceph grafana.a for f59f36e2-e80a-11ee-95cd-87774f69a715. 2024-03-22T05:28:51.962 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:28:51 smithi186 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@grafana.a.service: Consumed 3.375s CPU time. 2024-03-22T05:28:51.962 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:28:51 smithi186 systemd[1]: Starting Ceph grafana.a for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:28:51.962 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:28:51 smithi186 podman[49865]: 2024-03-22 05:28:51.839339104 +0000 UTC m=+0.017629357 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-03-22T05:28:51.962 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:28:51 smithi186 podman[49865]: 2024-03-22 05:28:51.962500555 +0000 UTC m=+0.140790792 container create a9629f797dd98668e49cc1df2a08d70eacfe6974e93247be2e8399a16e8d92d0 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a, summary=Grafana Container configured for Ceph mgr/dashboard integration, vendor=Red Hat, Inc., io.openshift.expose-services=, release=236.1648460182, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-03-28T10:36:18.413762, io.openshift.tags=base rhel8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.buildah.version=1.24.2, distribution-scope=public, architecture=x86_64, description=Ceph Grafana Container, name=ubi8, io.k8s.display-name=Red Hat Universal Base Image 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=ubi8-container, version=8.5, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, maintainer=Paul Cuzner ) 2024-03-22T05:28:52.219 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:28:52 smithi186 podman[49865]: 2024-03-22 05:28:52.190676181 +0000 UTC m=+0.368966422 container init a9629f797dd98668e49cc1df2a08d70eacfe6974e93247be2e8399a16e8d92d0 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a, vcs-type=git, version=8.5, com.redhat.component=ubi8-container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, release=236.1648460182, build-date=2022-03-28T10:36:18.413762, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=Ceph Grafana Container, maintainer=Paul Cuzner , distribution-scope=public, architecture=x86_64, io.k8s.display-name=Red Hat Universal Base Image 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.24.2, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, summary=Grafana Container configured for Ceph mgr/dashboard integration, vendor=Red Hat, Inc., io.openshift.tags=base rhel8, name=ubi8) 2024-03-22T05:28:52.219 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:28:52 smithi186 podman[49865]: 2024-03-22 05:28:52.193401048 +0000 UTC m=+0.371691287 container start a9629f797dd98668e49cc1df2a08d70eacfe6974e93247be2e8399a16e8d92d0 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a, io.openshift.expose-services=, vcs-type=git, summary=Grafana Container configured for Ceph mgr/dashboard integration, build-date=2022-03-28T10:36:18.413762, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=Red Hat Universal Base Image 8, maintainer=Paul Cuzner , architecture=x86_64, release=236.1648460182, version=8.5, description=Ceph Grafana Container, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, com.redhat.component=ubi8-container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.tags=base rhel8, name=ubi8, io.buildah.version=1.24.2, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, distribution-scope=public) 2024-03-22T05:28:52.471 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:28:52 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[49879]: Failed to start grafana. error: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-03-22T05:28:52.471 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:28:52 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[49879]: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-03-22T05:28:52.471 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:28:52 smithi186 bash[49865]: a9629f797dd98668e49cc1df2a08d70eacfe6974e93247be2e8399a16e8d92d0 2024-03-22T05:28:52.472 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:28:52 smithi186 systemd[1]: Started Ceph grafana.a for f59f36e2-e80a-11ee-95cd-87774f69a715. 2024-03-22T05:28:52.472 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:28:52 smithi186 podman[49895]: 2024-03-22 05:28:52.299357475 +0000 UTC m=+0.071993689 container died a9629f797dd98668e49cc1df2a08d70eacfe6974e93247be2e8399a16e8d92d0 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a, vendor=Red Hat, Inc., release=236.1648460182, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.k8s.display-name=Red Hat Universal Base Image 8, summary=Grafana Container configured for Ceph mgr/dashboard integration, version=8.5, vcs-type=git, build-date=2022-03-28T10:36:18.413762, maintainer=Paul Cuzner , com.redhat.component=ubi8-container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.buildah.version=1.24.2, description=Ceph Grafana Container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, io.openshift.tags=base rhel8, name=ubi8, io.openshift.expose-services=, architecture=x86_64) 2024-03-22T05:28:52.472 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:52 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: [22/Mar/2024:05:28:52] ENGINE Bus STOPPING 2024-03-22T05:28:52.472 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:52 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: [22/Mar/2024:05:28:52] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-22T05:28:52.472 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:52 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: [22/Mar/2024:05:28:52] ENGINE Bus STOPPED 2024-03-22T05:28:52.472 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:52 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: [22/Mar/2024:05:28:52] ENGINE Bus STARTING 2024-03-22T05:28:52.472 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:52 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: [22/Mar/2024:05:28:52] ENGINE Serving on http://:::9283 2024-03-22T05:28:52.472 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:52 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: [22/Mar/2024:05:28:52] ENGINE Bus STARTED 2024-03-22T05:28:52.473 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:52 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: [22/Mar/2024:05:28:52] ENGINE Bus STOPPING 2024-03-22T05:28:52.473 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:52 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: [22/Mar/2024:05:28:52] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-22T05:28:52.473 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:52 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: [22/Mar/2024:05:28:52] ENGINE Bus STOPPED 2024-03-22T05:28:52.764 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:52 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: [22/Mar/2024:05:28:52] ENGINE Bus STARTING 2024-03-22T05:28:52.765 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:52 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: [22/Mar/2024:05:28:52] ENGINE Serving on http://:::9283 2024-03-22T05:28:52.765 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:52 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: [22/Mar/2024:05:28:52] ENGINE Bus STARTED 2024-03-22T05:28:52.765 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:52 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: [22/Mar/2024:05:28:52] ENGINE Bus STOPPING 2024-03-22T05:28:52.765 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:52 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: [22/Mar/2024:05:28:52] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-22T05:28:52.765 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:52 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: [22/Mar/2024:05:28:52] ENGINE Bus STOPPED 2024-03-22T05:28:52.765 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:52 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: [22/Mar/2024:05:28:52] ENGINE Bus STARTING 2024-03-22T05:28:52.765 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:52 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: [22/Mar/2024:05:28:52] ENGINE Serving on http://:::9283 2024-03-22T05:28:52.765 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:52 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: [22/Mar/2024:05:28:52] ENGINE Bus STARTED 2024-03-22T05:28:53.043 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:28:52 smithi186 podman[49895]: 2024-03-22 05:28:52.894009288 +0000 UTC m=+0.666645503 container remove a9629f797dd98668e49cc1df2a08d70eacfe6974e93247be2e8399a16e8d92d0 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, architecture=x86_64, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, release=236.1648460182, build-date=2022-03-28T10:36:18.413762, vcs-type=git, distribution-scope=public, io.buildah.version=1.24.2, version=8.5, description=Ceph Grafana Container, io.openshift.tags=base rhel8, com.redhat.component=ubi8-container, maintainer=Paul Cuzner , summary=Grafana Container configured for Ceph mgr/dashboard integration, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., name=ubi8, io.openshift.expose-services=, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI) 2024-03-22T05:28:53.044 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:28:52 smithi186 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@grafana.a.service: Main process exited, code=exited, status=1/FAILURE 2024-03-22T05:28:53.044 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:28:52 smithi186 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-03-22T05:28:53.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:52 smithi186 ceph-mon[25720]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:28:53.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:52 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:53.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:52 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:53.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:52 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-22T05:28:53.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:52 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi005.front.sepia.ceph.com:9093"}]: dispatch 2024-03-22T05:28:53.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:52 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:53.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:52 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-22T05:28:53.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:52 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi186.front.sepia.ceph.com:9095"}]: dispatch 2024-03-22T05:28:53.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:52 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:53.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:52 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-22T05:28:53.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:52 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi186.front.sepia.ceph.com:3000"}]: dispatch 2024-03-22T05:28:53.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:52 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:53.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:52 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:28:53.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:52 smithi005 ceph-mon[24793]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:28:53.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:52 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:53.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:52 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:53.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:52 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-22T05:28:53.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:52 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi005.front.sepia.ceph.com:9093"}]: dispatch 2024-03-22T05:28:53.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:52 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:53.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:52 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-22T05:28:53.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:52 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi186.front.sepia.ceph.com:9095"}]: dispatch 2024-03-22T05:28:53.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:52 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:53.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:52 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-22T05:28:53.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:52 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi186.front.sepia.ceph.com:3000"}]: dispatch 2024-03-22T05:28:53.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:52 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:53.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:52 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:28:53.204 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:52 smithi005 ceph-mon[29523]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:28:53.204 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:52 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:53.204 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:52 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:53.204 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:52 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-22T05:28:53.204 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:52 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi005.front.sepia.ceph.com:9093"}]: dispatch 2024-03-22T05:28:53.204 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:52 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:53.204 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:52 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-22T05:28:53.204 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:52 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi186.front.sepia.ceph.com:9095"}]: dispatch 2024-03-22T05:28:53.204 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:52 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:53.204 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:52 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-22T05:28:53.204 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:52 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi186.front.sepia.ceph.com:3000"}]: dispatch 2024-03-22T05:28:53.204 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:52 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:53.204 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:52 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:28:53.544 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:28:53 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:28:53.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi005\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi005\", job=\"node\", machine=\"x86_64\", nodename=\"smithi005\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"f59f36e2-e80a-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi005\", job=\"node\", machine=\"x86_64\", nodename=\"smithi005\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-22T05:28:54.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:53 smithi186 ceph-mon[25720]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-22T05:28:54.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:53 smithi186 ceph-mon[25720]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi005.front.sepia.ceph.com:9093"}]: dispatch 2024-03-22T05:28:54.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:53 smithi186 ceph-mon[25720]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-22T05:28:54.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:53 smithi186 ceph-mon[25720]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi186.front.sepia.ceph.com:9095"}]: dispatch 2024-03-22T05:28:54.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:53 smithi186 ceph-mon[25720]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-22T05:28:54.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:53 smithi186 ceph-mon[25720]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi186.front.sepia.ceph.com:3000"}]: dispatch 2024-03-22T05:28:54.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:53 smithi005 ceph-mon[24793]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-22T05:28:54.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:53 smithi005 ceph-mon[24793]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi005.front.sepia.ceph.com:9093"}]: dispatch 2024-03-22T05:28:54.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:53 smithi005 ceph-mon[24793]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-22T05:28:54.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:53 smithi005 ceph-mon[24793]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi186.front.sepia.ceph.com:9095"}]: dispatch 2024-03-22T05:28:54.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:53 smithi005 ceph-mon[24793]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-22T05:28:54.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:53 smithi005 ceph-mon[24793]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi186.front.sepia.ceph.com:3000"}]: dispatch 2024-03-22T05:28:54.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:53 smithi005 ceph-mon[29523]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-22T05:28:54.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:53 smithi005 ceph-mon[29523]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi005.front.sepia.ceph.com:9093"}]: dispatch 2024-03-22T05:28:54.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:53 smithi005 ceph-mon[29523]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-22T05:28:54.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:53 smithi005 ceph-mon[29523]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi186.front.sepia.ceph.com:9095"}]: dispatch 2024-03-22T05:28:54.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:53 smithi005 ceph-mon[29523]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-22T05:28:54.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:53 smithi005 ceph-mon[29523]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi186.front.sepia.ceph.com:3000"}]: dispatch 2024-03-22T05:28:55.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:54 smithi186 ceph-mon[25720]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:28:55.116 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:54 smithi005 ceph-mon[29523]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:28:55.116 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:54 smithi005 ceph-mon[24793]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:28:57.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:56 smithi186 ceph-mon[25720]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 860 B/s rd, 0 op/s 2024-03-22T05:28:57.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:56 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:57.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:56 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:57.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:56 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:57.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:56 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:57.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:56 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:28:57.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:56 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:28:57.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:56 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:57.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:56 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:28:57.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:56 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:28:57.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:56 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:28:57.114 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:56 smithi005 ceph-mon[24793]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 860 B/s rd, 0 op/s 2024-03-22T05:28:57.115 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:56 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:57.115 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:56 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:57.115 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:56 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:57.115 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:56 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:57.115 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:56 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:28:57.115 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:56 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:28:57.115 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:56 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:57.115 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:56 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:28:57.115 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:56 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:28:57.115 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:56 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:28:57.116 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:56 smithi005 ceph-mon[29523]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 860 B/s rd, 0 op/s 2024-03-22T05:28:57.116 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:56 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:57.116 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:56 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:57.116 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:56 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:57.116 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:56 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:57.116 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:56 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:28:57.116 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:56 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:28:57.116 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:56 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:28:57.116 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:56 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:28:57.117 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:56 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:28:57.117 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:56 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:28:58.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:57 smithi186 ceph-mon[25720]: from='client.24991 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:28:58.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:57 smithi186 ceph-mon[25720]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-22T05:28:58.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:57 smithi186 ceph-mon[25720]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:28:58.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:57 smithi186 ceph-mon[25720]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:28:58.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:57 smithi186 ceph-mon[25720]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-22T05:28:58.052 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:57 smithi005 ceph-mon[24793]: from='client.24991 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:28:58.052 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:57 smithi005 ceph-mon[24793]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-22T05:28:58.052 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:57 smithi005 ceph-mon[24793]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:28:58.052 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:57 smithi005 ceph-mon[24793]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:28:58.052 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:57 smithi005 ceph-mon[24793]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-22T05:28:58.053 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:57 smithi005 ceph-mon[29523]: from='client.24991 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:28:58.053 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:57 smithi005 ceph-mon[29523]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-22T05:28:58.053 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:57 smithi005 ceph-mon[29523]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:28:58.053 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:57 smithi005 ceph-mon[29523]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:28:58.053 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:57 smithi005 ceph-mon[29523]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-22T05:28:59.046 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:28:58 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:28:58.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.5\", device_class=\"ssd\", hostname=\"smithi005\", instance=\"172.21.15.186:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.5\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"f59f36e2-e80a-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.5\", device_class=\"ssd\", hostname=\"smithi005\", instance=\"smithi186:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.5\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-22T05:28:59.502 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:28:59 smithi005 ceph-mon[29523]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 951 B/s rd, 0 op/s 2024-03-22T05:28:59.502 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:28:59 smithi005 ceph-mon[24793]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 951 B/s rd, 0 op/s 2024-03-22T05:28:59.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:28:59 smithi186 ceph-mon[25720]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 951 B/s rd, 0 op/s 2024-03-22T05:29:00.294 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:59 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: 2024-03-22T05:28:59.867+0000 7f0393508700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:29:00.294 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:59 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: Creating ceph-iscsi config... 2024-03-22T05:29:00.294 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:59 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:29:00.294 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:59 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:29:00.294 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:59 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:29:00.294 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:59 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:29:00.295 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:59 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:29:00.295 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:59 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: Traceback (most recent call last): 2024-03-22T05:29:00.295 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:59 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:29:00.295 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:59 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: return _run_code(code, main_globals, None, 2024-03-22T05:29:00.295 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:59 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:29:00.295 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:59 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: exec(code, run_globals) 2024-03-22T05:29:00.295 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:59 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:29:00.295 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:59 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:29:00.295 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:59 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:29:00.295 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:59 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:29:00.295 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:59 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:29:00.295 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:59 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:29:00.295 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:59 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:29:00.296 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:59 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:29:00.296 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:59 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:29:00.296 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:59 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: Traceback (most recent call last): 2024-03-22T05:29:00.296 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:59 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:29:00.296 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:59 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:29:00.296 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:59 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:29:00.296 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:59 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:29:00.297 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:59 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:29:00.297 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:59 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:29:00.297 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:59 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:29:00.297 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:59 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: return future.result(timeout) 2024-03-22T05:29:00.297 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:59 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:29:00.297 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:59 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: return self.__get_result() 2024-03-22T05:29:00.297 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:59 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:29:00.297 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:59 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: raise self._exception 2024-03-22T05:29:00.297 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:59 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:29:00.297 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:59 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: config_blobs=daemon_spec.final_config, 2024-03-22T05:29:00.297 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:59 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:29:00.297 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:59 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:29:00.297 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:59 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:29:00.297 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:59 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: Creating ceph-iscsi config... 2024-03-22T05:29:00.297 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:59 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:29:00.298 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:59 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:29:00.298 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:59 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:29:00.298 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:59 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:29:00.298 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:59 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:29:00.298 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:59 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: Traceback (most recent call last): 2024-03-22T05:29:00.298 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:59 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:29:00.298 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:59 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: return _run_code(code, main_globals, None, 2024-03-22T05:29:00.298 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:59 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:29:00.298 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:59 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: exec(code, run_globals) 2024-03-22T05:29:00.298 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:59 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:29:00.299 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:59 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:29:00.299 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:59 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:29:00.299 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:59 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:29:00.299 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:59 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:29:00.299 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:59 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:29:00.299 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:59 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:29:00.299 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:59 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:29:00.299 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:28:59 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:29:01.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:29:00 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:29:01.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:29:00 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:29:01.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:29:00 smithi005 ceph-mon[29523]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:29:01.203 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:29:01.203 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:29:01.203 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:29:01.203 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:29:01.203 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:29:01.203 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:29:01.203 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:29:01.203 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:29:01.203 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:29:01.203 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:29:01.203 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:29:01.203 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:29:01.204 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:29:01.204 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:29:01.204 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:29:01.204 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:29:01.204 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:29:01.204 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:29:01.204 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:29:01.204 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:29:01.204 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:29:01.204 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:29:01.204 INFO:journalctl@ceph.mon.c.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:29:01.204 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:29:01.204 INFO:journalctl@ceph.mon.c.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:29:01.204 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:29:01.205 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:29:01.205 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:29:01.205 INFO:journalctl@ceph.mon.c.smithi005.stdout: return future.result(timeout) 2024-03-22T05:29:01.205 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:29:01.205 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.__get_result() 2024-03-22T05:29:01.205 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:29:01.205 INFO:journalctl@ceph.mon.c.smithi005.stdout: raise self._exception 2024-03-22T05:29:01.205 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:29:01.205 INFO:journalctl@ceph.mon.c.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:29:01.205 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:29:01.205 INFO:journalctl@ceph.mon.c.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:29:01.205 INFO:journalctl@ceph.mon.c.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:29:01.205 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:29:01.206 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:29:01.206 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:29:01.206 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:29:01.206 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:29:01.206 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:29:01.206 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:29:01.206 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:29:01.206 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:29:01.206 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:29:01.206 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:29:01.206 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:29:01.206 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:29:01.207 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:29:01.207 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:29:00 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:29:01.207 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:29:00 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:29:01.207 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:29:00 smithi005 ceph-mon[24793]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:29:01.207 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:29:01.207 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:29:01.207 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:29:01.207 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:29:01.207 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:29:01.208 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:29:01.208 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:29:01.208 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:29:01.208 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:29:01.208 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:29:01.208 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:29:01.208 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:29:01.208 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:29:01.208 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:29:01.208 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:29:01.208 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:29:01.208 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:29:01.208 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:29:01.209 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:29:01.209 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:29:01.209 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:29:01.209 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:29:01.209 INFO:journalctl@ceph.mon.a.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:29:01.209 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:29:01.210 INFO:journalctl@ceph.mon.a.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:29:01.210 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:29:01.210 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:29:01.210 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:29:01.210 INFO:journalctl@ceph.mon.a.smithi005.stdout: return future.result(timeout) 2024-03-22T05:29:01.210 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:29:01.210 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.__get_result() 2024-03-22T05:29:01.211 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:29:01.211 INFO:journalctl@ceph.mon.a.smithi005.stdout: raise self._exception 2024-03-22T05:29:01.211 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:29:01.211 INFO:journalctl@ceph.mon.a.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:29:01.211 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:29:01.211 INFO:journalctl@ceph.mon.a.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:29:01.211 INFO:journalctl@ceph.mon.a.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:29:01.211 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:29:01.211 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:29:01.211 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:29:01.211 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:29:01.212 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:29:01.212 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:29:01.212 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:29:01.212 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:29:01.212 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:29:01.212 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:29:01.212 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:29:01.212 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:29:01.212 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:29:01.212 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:29:01.212 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:29:01.212 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:29:01.212 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:29:01.213 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:29:01.213 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:29:01.213 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:29:01.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:29:00 smithi005 ceph-mon[24793]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 951 B/s rd, 0 op/s 2024-03-22T05:29:01.213 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:29:01.213 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:29:01.213 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:29:01.213 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:29:01.213 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:29:01.214 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:29:01.214 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:29:00 smithi005 ceph-mon[29523]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 951 B/s rd, 0 op/s 2024-03-22T05:29:01.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:29:00 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:29:01.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:29:00 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:29:01.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:29:00 smithi186 ceph-mon[25720]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:29:01.294 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:29:01.294 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:29:01.294 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:29:01.294 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:29:01.294 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:29:01.294 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:29:01.294 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:29:01.294 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:29:01.295 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:29:01.295 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:29:01.295 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:29:01.295 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:29:01.295 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:29:01.295 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:29:01.295 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:29:01.295 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:29:01.295 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:29:01.295 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:29:01.295 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:29:01.295 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:29:01.295 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:29:01.295 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:29:01.296 INFO:journalctl@ceph.mon.b.smithi186.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:29:01.296 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:29:01.296 INFO:journalctl@ceph.mon.b.smithi186.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:29:01.296 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:29:01.296 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:29:01.296 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:29:01.296 INFO:journalctl@ceph.mon.b.smithi186.stdout: return future.result(timeout) 2024-03-22T05:29:01.296 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:29:01.296 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.__get_result() 2024-03-22T05:29:01.296 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:29:01.296 INFO:journalctl@ceph.mon.b.smithi186.stdout: raise self._exception 2024-03-22T05:29:01.296 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:29:01.296 INFO:journalctl@ceph.mon.b.smithi186.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:29:01.296 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:29:01.296 INFO:journalctl@ceph.mon.b.smithi186.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:29:01.296 INFO:journalctl@ceph.mon.b.smithi186.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:29:01.297 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:29:01.297 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:29:01.297 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:29:01.297 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:29:01.297 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:29:01.297 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:29:01.297 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:29:01.297 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:29:01.297 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:29:01.297 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:29:01.297 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:29:01.297 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:29:01.297 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:29:01.298 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:29:01.298 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:29:01.298 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:29:01.298 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:29:01.298 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:29:01.298 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:29:01.298 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:29:01.298 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:29:00 smithi186 ceph-mon[25720]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 951 B/s rd, 0 op/s 2024-03-22T05:29:01.793 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:29:01 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:29:01] "GET /metrics HTTP/1.1" 200 37955 "" "Prometheus/2.33.4" 2024-03-22T05:29:03.310 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:29:03 smithi186 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@grafana.a.service: Scheduled restart job, restart counter is at 1. 2024-03-22T05:29:03.310 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:29:03 smithi186 systemd[1]: Stopped Ceph grafana.a for f59f36e2-e80a-11ee-95cd-87774f69a715. 2024-03-22T05:29:03.310 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:29:03 smithi186 systemd[1]: Starting Ceph grafana.a for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:29:03.310 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:29:03 smithi186 podman[50553]: 2024-03-22 05:29:03.18462466 +0000 UTC m=+0.017545350 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-03-22T05:29:03.576 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:29:03 smithi186 ceph-mon[25720]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 856 B/s rd, 0 op/s 2024-03-22T05:29:03.576 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:29:03 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:29:03.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi005\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi005\", job=\"node\", machine=\"x86_64\", nodename=\"smithi005\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"f59f36e2-e80a-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi005\", job=\"node\", machine=\"x86_64\", nodename=\"smithi005\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-22T05:29:03.577 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:29:03 smithi186 podman[50553]: 2024-03-22 05:29:03.310670757 +0000 UTC m=+0.143591437 container create a2879810efa5c6a40bf92b4a4f3ab1997b5010f3c66650a020108eefbcf569f7 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a, summary=Grafana Container configured for Ceph mgr/dashboard integration, build-date=2022-03-28T10:36:18.413762, architecture=x86_64, com.redhat.component=ubi8-container, vcs-type=git, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.tags=base rhel8, release=236.1648460182, io.k8s.display-name=Red Hat Universal Base Image 8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8.5, description=Ceph Grafana Container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=ubi8, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, maintainer=Paul Cuzner , distribution-scope=public, vendor=Red Hat, Inc., io.buildah.version=1.24.2) 2024-03-22T05:29:03.577 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:29:03 smithi186 podman[50553]: 2024-03-22 05:29:03.546449868 +0000 UTC m=+0.379370555 container init a2879810efa5c6a40bf92b4a4f3ab1997b5010f3c66650a020108eefbcf569f7 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vendor=Red Hat, Inc., release=236.1648460182, description=Ceph Grafana Container, maintainer=Paul Cuzner , build-date=2022-03-28T10:36:18.413762, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, version=8.5, summary=Grafana Container configured for Ceph mgr/dashboard integration, name=ubi8, io.openshift.expose-services=, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.component=ubi8-container, io.openshift.tags=base rhel8, io.buildah.version=1.24.2, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, distribution-scope=public) 2024-03-22T05:29:03.577 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:29:03 smithi186 podman[50553]: 2024-03-22 05:29:03.549153087 +0000 UTC m=+0.382073773 container start a2879810efa5c6a40bf92b4a4f3ab1997b5010f3c66650a020108eefbcf569f7 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a, name=ubi8, description=Ceph Grafana Container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vcs-type=git, release=236.1648460182, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=ubi8-container, build-date=2022-03-28T10:36:18.413762, io.openshift.tags=base rhel8, io.openshift.expose-services=, io.buildah.version=1.24.2, architecture=x86_64, distribution-scope=public, vendor=Red Hat, Inc., summary=Grafana Container configured for Ceph mgr/dashboard integration, io.k8s.display-name=Red Hat Universal Base Image 8, maintainer=Paul Cuzner , com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, version=8.5) 2024-03-22T05:29:03.577 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:29:03 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[50567]: Failed to start grafana. error: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-03-22T05:29:03.577 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:29:03 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[50567]: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-03-22T05:29:03.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:29:03 smithi005 ceph-mon[24793]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 856 B/s rd, 0 op/s 2024-03-22T05:29:03.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:29:03 smithi005 ceph-mon[29523]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 856 B/s rd, 0 op/s 2024-03-22T05:29:04.043 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:29:03 smithi186 bash[50553]: a2879810efa5c6a40bf92b4a4f3ab1997b5010f3c66650a020108eefbcf569f7 2024-03-22T05:29:04.044 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:29:03 smithi186 systemd[1]: Started Ceph grafana.a for f59f36e2-e80a-11ee-95cd-87774f69a715. 2024-03-22T05:29:04.044 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:29:03 smithi186 podman[50583]: 2024-03-22 05:29:03.656864387 +0000 UTC m=+0.073052470 container died a2879810efa5c6a40bf92b4a4f3ab1997b5010f3c66650a020108eefbcf569f7 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a, version=8.5, distribution-scope=public, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, name=ubi8, build-date=2022-03-28T10:36:18.413762, description=Ceph Grafana Container, maintainer=Paul Cuzner , vcs-type=git, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.tags=base rhel8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=ubi8-container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.24.2, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.expose-services=, release=236.1648460182, architecture=x86_64) 2024-03-22T05:29:04.544 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:29:04 smithi186 podman[50583]: 2024-03-22 05:29:04.183604447 +0000 UTC m=+0.599792569 container remove a2879810efa5c6a40bf92b4a4f3ab1997b5010f3c66650a020108eefbcf569f7 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a, io.buildah.version=1.24.2, name=ubi8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.component=ubi8-container, vcs-type=git, release=236.1648460182, description=Ceph Grafana Container, io.openshift.tags=base rhel8, distribution-scope=public, version=8.5, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Grafana Container configured for Ceph mgr/dashboard integration, maintainer=Paul Cuzner , com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, architecture=x86_64, vendor=Red Hat, Inc., build-date=2022-03-28T10:36:18.413762, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=Red Hat Universal Base Image 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182) 2024-03-22T05:29:04.544 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:29:04 smithi186 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@grafana.a.service: Main process exited, code=exited, status=1/FAILURE 2024-03-22T05:29:04.544 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:29:04 smithi186 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-03-22T05:29:05.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:29:05 smithi186 ceph-mon[25720]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 856 B/s rd, 0 op/s 2024-03-22T05:29:05.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:29:05 smithi005 ceph-mon[29523]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 856 B/s rd, 0 op/s 2024-03-22T05:29:05.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:29:05 smithi005 ceph-mon[24793]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 856 B/s rd, 0 op/s 2024-03-22T05:29:07.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:29:07 smithi186 ceph-mon[25720]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 856 B/s rd, 0 op/s 2024-03-22T05:29:07.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:29:07 smithi005 ceph-mon[29523]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 856 B/s rd, 0 op/s 2024-03-22T05:29:07.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:29:07 smithi005 ceph-mon[24793]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 856 B/s rd, 0 op/s 2024-03-22T05:29:09.044 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:29:08 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:29:08.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.5\", device_class=\"ssd\", hostname=\"smithi005\", instance=\"172.21.15.186:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.5\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"f59f36e2-e80a-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.5\", device_class=\"ssd\", hostname=\"smithi005\", instance=\"smithi186:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.5\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-22T05:29:09.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:29:09 smithi186 ceph-mon[25720]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-22T05:29:09.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:29:09 smithi005 ceph-mon[29523]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-22T05:29:09.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:29:09 smithi005 ceph-mon[24793]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-22T05:29:11.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:29:11 smithi186 ceph-mon[25720]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-22T05:29:11.794 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:29:11 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:29:11] "GET /metrics HTTP/1.1" 200 37955 "" "Prometheus/2.33.4" 2024-03-22T05:29:11.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:29:11 smithi005 ceph-mon[24793]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-22T05:29:11.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:29:11 smithi005 ceph-mon[29523]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-22T05:29:12.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:29:12 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/4284618275' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-22T05:29:12.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:29:12 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/4284618275' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-22T05:29:12.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:29:12 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/4284618275' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-22T05:29:13.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:29:13 smithi186 ceph-mon[25720]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-22T05:29:13.794 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:29:13 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:29:13.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi005\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi005\", job=\"node\", machine=\"x86_64\", nodename=\"smithi005\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"f59f36e2-e80a-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi005\", job=\"node\", machine=\"x86_64\", nodename=\"smithi005\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-22T05:29:13.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:29:13 smithi005 ceph-mon[24793]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-22T05:29:13.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:29:13 smithi005 ceph-mon[29523]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-22T05:29:14.560 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:29:14 smithi186 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@grafana.a.service: Scheduled restart job, restart counter is at 2. 2024-03-22T05:29:14.561 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:29:14 smithi186 systemd[1]: Stopped Ceph grafana.a for f59f36e2-e80a-11ee-95cd-87774f69a715. 2024-03-22T05:29:14.561 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:29:14 smithi186 systemd[1]: Starting Ceph grafana.a for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:29:14.561 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:29:14 smithi186 podman[50654]: 2024-03-22 05:29:14.432778029 +0000 UTC m=+0.018672025 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-03-22T05:29:14.847 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:29:14 smithi186 podman[50654]: 2024-03-22 05:29:14.561658661 +0000 UTC m=+0.147552699 container create 91afb580592b5b6ff744ae865e14eade16ff8017cf4ff0f29218d067082fa0f7 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a, io.openshift.tags=base rhel8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=236.1648460182, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, distribution-scope=public, vendor=Red Hat, Inc., version=8.5, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Paul Cuzner , description=Ceph Grafana Container, name=ubi8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.buildah.version=1.24.2, io.openshift.expose-services=, build-date=2022-03-28T10:36:18.413762, com.redhat.component=ubi8-container, architecture=x86_64, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-type=git, io.k8s.display-name=Red Hat Universal Base Image 8) 2024-03-22T05:29:14.847 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:29:14 smithi186 podman[50654]: 2024-03-22 05:29:14.763590734 +0000 UTC m=+0.349484740 container init 91afb580592b5b6ff744ae865e14eade16ff8017cf4ff0f29218d067082fa0f7 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, distribution-scope=public, com.redhat.component=ubi8-container, io.k8s.display-name=Red Hat Universal Base Image 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, build-date=2022-03-28T10:36:18.413762, vendor=Red Hat, Inc., name=ubi8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Paul Cuzner , io.openshift.tags=base rhel8, description=Ceph Grafana Container, architecture=x86_64, summary=Grafana Container configured for Ceph mgr/dashboard integration, version=8.5, io.buildah.version=1.24.2, release=236.1648460182) 2024-03-22T05:29:14.847 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:29:14 smithi186 podman[50654]: 2024-03-22 05:29:14.766210611 +0000 UTC m=+0.352104610 container start 91afb580592b5b6ff744ae865e14eade16ff8017cf4ff0f29218d067082fa0f7 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a, io.openshift.tags=base rhel8, name=ubi8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.k8s.display-name=Red Hat Universal Base Image 8, maintainer=Paul Cuzner , io.openshift.expose-services=, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.component=ubi8-container, vcs-type=git, release=236.1648460182, build-date=2022-03-28T10:36:18.413762, architecture=x86_64, io.buildah.version=1.24.2, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., description=Ceph Grafana Container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Grafana Container configured for Ceph mgr/dashboard integration, distribution-scope=public, version=8.5) 2024-03-22T05:29:14.847 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:29:14 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[50668]: Failed to start grafana. error: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-03-22T05:29:14.847 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:29:14 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[50668]: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-03-22T05:29:14.847 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:29:14 smithi186 bash[50654]: 91afb580592b5b6ff744ae865e14eade16ff8017cf4ff0f29218d067082fa0f7 2024-03-22T05:29:15.134 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:29:14 smithi186 systemd[1]: Started Ceph grafana.a for f59f36e2-e80a-11ee-95cd-87774f69a715. 2024-03-22T05:29:15.134 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:29:14 smithi186 podman[50684]: 2024-03-22 05:29:14.87544125 +0000 UTC m=+0.075739125 container died 91afb580592b5b6ff744ae865e14eade16ff8017cf4ff0f29218d067082fa0f7 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a, architecture=x86_64, description=Ceph Grafana Container, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base rhel8, summary=Grafana Container configured for Ceph mgr/dashboard integration, version=8.5, release=236.1648460182, vcs-type=git, maintainer=Paul Cuzner , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, com.redhat.component=ubi8-container, io.buildah.version=1.24.2, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, name=ubi8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.expose-services=, io.k8s.display-name=Red Hat Universal Base Image 8, vendor=Red Hat, Inc., build-date=2022-03-28T10:36:18.413762, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-03-22T05:29:15.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:29:15 smithi186 ceph-mon[25720]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:29:15.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:29:15 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:29:15.794 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:29:15 smithi186 podman[50684]: 2024-03-22 05:29:15.426129746 +0000 UTC m=+0.626427604 container remove 91afb580592b5b6ff744ae865e14eade16ff8017cf4ff0f29218d067082fa0f7 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a, architecture=x86_64, name=ubi8, vcs-type=git, vendor=Red Hat, Inc., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, release=236.1648460182, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8.5, maintainer=Paul Cuzner , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.buildah.version=1.24.2, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.expose-services=, build-date=2022-03-28T10:36:18.413762, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.component=ubi8-container, io.openshift.tags=base rhel8, description=Ceph Grafana Container) 2024-03-22T05:29:15.794 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:29:15 smithi186 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@grafana.a.service: Main process exited, code=exited, status=1/FAILURE 2024-03-22T05:29:15.794 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:29:15 smithi186 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-03-22T05:29:15.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:29:15 smithi005 ceph-mon[24793]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:29:15.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:29:15 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:29:15.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:29:15 smithi005 ceph-mon[29523]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:29:15.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:29:15 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:29:17.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:29:17 smithi186 ceph-mon[25720]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-03-22T05:29:17.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:29:17 smithi005 ceph-mon[24793]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-03-22T05:29:17.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:29:17 smithi005 ceph-mon[29523]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-03-22T05:29:19.044 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:29:18 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:29:18.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.5\", device_class=\"ssd\", hostname=\"smithi005\", instance=\"172.21.15.186:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.5\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"f59f36e2-e80a-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.5\", device_class=\"ssd\", hostname=\"smithi005\", instance=\"smithi186:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.5\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-22T05:29:19.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:29:19 smithi186 ceph-mon[25720]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-03-22T05:29:19.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:29:19 smithi005 ceph-mon[24793]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-03-22T05:29:19.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:29:19 smithi005 ceph-mon[29523]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-03-22T05:29:21.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:29:21 smithi186 ceph-mon[25720]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-03-22T05:29:21.794 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:29:21 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:29:21] "GET /metrics HTTP/1.1" 200 37955 "" "Prometheus/2.33.4" 2024-03-22T05:29:21.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:29:21 smithi005 ceph-mon[29523]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-03-22T05:29:21.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:29:21 smithi005 ceph-mon[24793]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-03-22T05:29:22.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:29:22 smithi186 ceph-mon[25720]: from='client.25054 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:29:22.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:29:22 smithi005 ceph-mon[29523]: from='client.25054 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:29:22.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:29:22 smithi005 ceph-mon[24793]: from='client.25054 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:29:23.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:29:23 smithi186 ceph-mon[25720]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:29:23.795 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:29:23 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:29:23.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi005\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi005\", job=\"node\", machine=\"x86_64\", nodename=\"smithi005\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"f59f36e2-e80a-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi005\", job=\"node\", machine=\"x86_64\", nodename=\"smithi005\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-22T05:29:23.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:29:23 smithi005 ceph-mon[24793]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:29:23.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:29:23 smithi005 ceph-mon[29523]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:29:25.771 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:29:25 smithi186 ceph-mon[25720]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:29:25.771 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:29:25 smithi186 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@grafana.a.service: Scheduled restart job, restart counter is at 3. 2024-03-22T05:29:25.771 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:29:25 smithi186 systemd[1]: Stopped Ceph grafana.a for f59f36e2-e80a-11ee-95cd-87774f69a715. 2024-03-22T05:29:25.771 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:29:25 smithi186 systemd[1]: Starting Ceph grafana.a for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:29:25.771 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:29:25 smithi186 podman[50767]: 2024-03-22 05:29:25.648816696 +0000 UTC m=+0.017665860 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-03-22T05:29:25.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:29:25 smithi005 ceph-mon[24793]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:29:25.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:29:25 smithi005 ceph-mon[29523]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:29:26.023 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:29:25 smithi186 podman[50767]: 2024-03-22 05:29:25.771753152 +0000 UTC m=+0.140602306 container create 5fb1e72cbc6810a39c2e5a83735d2824aa84f96023a51c04d10492c6814ee985 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a, maintainer=Paul Cuzner , version=8.5, summary=Grafana Container configured for Ceph mgr/dashboard integration, release=236.1648460182, com.redhat.component=ubi8-container, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.openshift.expose-services=, name=ubi8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-type=git, io.k8s.display-name=Red Hat Universal Base Image 8, io.buildah.version=1.24.2, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base rhel8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, build-date=2022-03-28T10:36:18.413762, vendor=Red Hat, Inc., architecture=x86_64, description=Ceph Grafana Container) 2024-03-22T05:29:26.294 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:29:26 smithi186 podman[50767]: 2024-03-22 05:29:26.024435936 +0000 UTC m=+0.393285101 container init 5fb1e72cbc6810a39c2e5a83735d2824aa84f96023a51c04d10492c6814ee985 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, build-date=2022-03-28T10:36:18.413762, description=Ceph Grafana Container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=236.1648460182, io.openshift.tags=base rhel8, io.k8s.display-name=Red Hat Universal Base Image 8, vendor=Red Hat, Inc., name=ubi8, io.buildah.version=1.24.2, io.openshift.expose-services=, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, distribution-scope=public, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.component=ubi8-container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, maintainer=Paul Cuzner , architecture=x86_64, version=8.5) 2024-03-22T05:29:26.294 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:29:26 smithi186 podman[50767]: 2024-03-22 05:29:26.027238171 +0000 UTC m=+0.396087330 container start 5fb1e72cbc6810a39c2e5a83735d2824aa84f96023a51c04d10492c6814ee985 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a, release=236.1648460182, name=ubi8, architecture=x86_64, summary=Grafana Container configured for Ceph mgr/dashboard integration, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.k8s.display-name=Red Hat Universal Base Image 8, version=8.5, io.openshift.expose-services=, vendor=Red Hat, Inc., maintainer=Paul Cuzner , vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, distribution-scope=public, build-date=2022-03-28T10:36:18.413762, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, com.redhat.component=ubi8-container, io.openshift.tags=base rhel8, description=Ceph Grafana Container, io.buildah.version=1.24.2, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-03-22T05:29:26.294 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:29:26 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[50781]: Failed to start grafana. error: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-03-22T05:29:26.294 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:29:26 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[50781]: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-03-22T05:29:26.294 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:29:26 smithi186 bash[50767]: 5fb1e72cbc6810a39c2e5a83735d2824aa84f96023a51c04d10492c6814ee985 2024-03-22T05:29:26.294 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:29:26 smithi186 systemd[1]: Started Ceph grafana.a for f59f36e2-e80a-11ee-95cd-87774f69a715. 2024-03-22T05:29:26.294 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:29:26 smithi186 podman[50797]: 2024-03-22 05:29:26.112766807 +0000 UTC m=+0.052963663 container died 5fb1e72cbc6810a39c2e5a83735d2824aa84f96023a51c04d10492c6814ee985 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a, description=Ceph Grafana Container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vcs-type=git, io.k8s.display-name=Red Hat Universal Base Image 8, version=8.5, release=236.1648460182, vendor=Red Hat, Inc., io.buildah.version=1.24.2, io.openshift.expose-services=, summary=Grafana Container configured for Ceph mgr/dashboard integration, maintainer=Paul Cuzner , distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, com.redhat.component=ubi8-container, build-date=2022-03-28T10:36:18.413762, io.openshift.tags=base rhel8, name=ubi8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182) 2024-03-22T05:29:27.043 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:29:26 smithi186 podman[50797]: 2024-03-22 05:29:26.803303893 +0000 UTC m=+0.743500750 container remove 5fb1e72cbc6810a39c2e5a83735d2824aa84f96023a51c04d10492c6814ee985 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, vendor=Red Hat, Inc., description=Ceph Grafana Container, release=236.1648460182, maintainer=Paul Cuzner , com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, version=8.5, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base rhel8, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.component=ubi8-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, distribution-scope=public, io.buildah.version=1.24.2, name=ubi8, vcs-type=git, build-date=2022-03-28T10:36:18.413762, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56) 2024-03-22T05:29:27.044 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:29:26 smithi186 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@grafana.a.service: Main process exited, code=exited, status=1/FAILURE 2024-03-22T05:29:27.044 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:29:26 smithi186 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-03-22T05:29:27.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:29:27 smithi005 ceph-mon[24793]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:29:27.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:29:27 smithi005 ceph-mon[29523]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:29:28.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:29:27 smithi186 ceph-mon[25720]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:29:29.044 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:29:28 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:29:28.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.5\", device_class=\"ssd\", hostname=\"smithi005\", instance=\"172.21.15.186:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.5\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"f59f36e2-e80a-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.5\", device_class=\"ssd\", hostname=\"smithi005\", instance=\"smithi186:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.5\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-22T05:29:29.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:29:29 smithi186 ceph-mon[25720]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:29:29.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:29:29 smithi005 ceph-mon[24793]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:29:29.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:29:29 smithi005 ceph-mon[29523]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:29:30.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:29:30 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:29:30.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:29:30 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:29:30.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:29:30 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:29:31.511 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:29:31 smithi186 ceph-mon[25720]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:29:31.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:29:31 smithi005 ceph-mon[24793]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:29:31.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:29:31 smithi005 ceph-mon[29523]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:29:31.793 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:29:31 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:29:31] "GET /metrics HTTP/1.1" 200 37951 "" "Prometheus/2.33.4" 2024-03-22T05:29:32.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:29:32 smithi186 ceph-mon[25720]: from='client.25054 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:29:32.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:29:32 smithi005 ceph-mon[24793]: from='client.25054 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:29:32.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:29:32 smithi005 ceph-mon[29523]: from='client.25054 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:29:33.503 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:29:33 smithi186 ceph-mon[25720]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:29:33.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:29:33 smithi005 ceph-mon[24793]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:29:33.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:29:33 smithi005 ceph-mon[29523]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:29:33.794 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:29:33 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:29:33.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi005\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi005\", job=\"node\", machine=\"x86_64\", nodename=\"smithi005\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"f59f36e2-e80a-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi005\", job=\"node\", machine=\"x86_64\", nodename=\"smithi005\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-22T05:29:35.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:29:35 smithi186 ceph-mon[25720]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:29:35.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:29:35 smithi005 ceph-mon[29523]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:29:35.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:29:35 smithi005 ceph-mon[24793]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:29:37.382 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:29:37 smithi186 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@grafana.a.service: Scheduled restart job, restart counter is at 4. 2024-03-22T05:29:37.382 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:29:37 smithi186 systemd[1]: Stopped Ceph grafana.a for f59f36e2-e80a-11ee-95cd-87774f69a715. 2024-03-22T05:29:37.383 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:29:37 smithi186 systemd[1]: Starting Ceph grafana.a for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:29:37.383 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:29:37 smithi186 podman[50864]: 2024-03-22 05:29:37.176646632 +0000 UTC m=+0.017500592 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-03-22T05:29:37.383 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:29:37 smithi186 podman[50864]: 2024-03-22 05:29:37.287188897 +0000 UTC m=+0.128042856 container create bd0e7dd9056c94da2c9c80bdcd99cf8418369635966a3572fc536e5bb97fa7c5 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a, com.redhat.component=ubi8-container, description=Ceph Grafana Container, build-date=2022-03-28T10:36:18.413762, io.buildah.version=1.24.2, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=236.1648460182, maintainer=Paul Cuzner , vendor=Red Hat, Inc., name=ubi8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, version=8.5, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.tags=base rhel8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, distribution-scope=public, io.k8s.display-name=Red Hat Universal Base Image 8, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-type=git, architecture=x86_64) 2024-03-22T05:29:37.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:29:37 smithi186 ceph-mon[25720]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:29:37.794 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:29:37 smithi186 podman[50864]: 2024-03-22 05:29:37.481278568 +0000 UTC m=+0.322132542 container init bd0e7dd9056c94da2c9c80bdcd99cf8418369635966a3572fc536e5bb97fa7c5 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a, description=Ceph Grafana Container, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, maintainer=Paul Cuzner , vendor=Red Hat, Inc., io.openshift.expose-services=, io.buildah.version=1.24.2, release=236.1648460182, architecture=x86_64, io.k8s.display-name=Red Hat Universal Base Image 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8.5, com.redhat.component=ubi8-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, name=ubi8, build-date=2022-03-28T10:36:18.413762, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-type=git, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.tags=base rhel8) 2024-03-22T05:29:37.794 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:29:37 smithi186 podman[50864]: 2024-03-22 05:29:37.483791505 +0000 UTC m=+0.324645467 container start bd0e7dd9056c94da2c9c80bdcd99cf8418369635966a3572fc536e5bb97fa7c5 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, io.openshift.tags=base rhel8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, maintainer=Paul Cuzner , description=Ceph Grafana Container, com.redhat.component=ubi8-container, build-date=2022-03-28T10:36:18.413762, vendor=Red Hat, Inc., name=ubi8, io.k8s.display-name=Red Hat Universal Base Image 8, vcs-type=git, summary=Grafana Container configured for Ceph mgr/dashboard integration, release=236.1648460182, architecture=x86_64, version=8.5, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.buildah.version=1.24.2, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-03-22T05:29:37.794 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:29:37 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[50879]: Failed to start grafana. error: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-03-22T05:29:37.794 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:29:37 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[50879]: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-03-22T05:29:37.794 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:29:37 smithi186 bash[50864]: bd0e7dd9056c94da2c9c80bdcd99cf8418369635966a3572fc536e5bb97fa7c5 2024-03-22T05:29:37.794 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:29:37 smithi186 systemd[1]: Started Ceph grafana.a for f59f36e2-e80a-11ee-95cd-87774f69a715. 2024-03-22T05:29:37.794 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:29:37 smithi186 podman[50895]: 2024-03-22 05:29:37.589241696 +0000 UTC m=+0.071955400 container died bd0e7dd9056c94da2c9c80bdcd99cf8418369635966a3572fc536e5bb97fa7c5 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.component=ubi8-container, description=Ceph Grafana Container, io.openshift.tags=base rhel8, version=8.5, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vendor=Red Hat, Inc., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.expose-services=, build-date=2022-03-28T10:36:18.413762, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=236.1648460182, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=ubi8, architecture=x86_64, maintainer=Paul Cuzner , io.k8s.display-name=Red Hat Universal Base Image 8, summary=Grafana Container configured for Ceph mgr/dashboard integration, distribution-scope=public, vcs-type=git, io.buildah.version=1.24.2) 2024-03-22T05:29:37.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:29:37 smithi005 ceph-mon[29523]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:29:37.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:29:37 smithi005 ceph-mon[24793]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:29:38.544 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:29:38 smithi186 podman[50895]: 2024-03-22 05:29:38.168760893 +0000 UTC m=+0.651474598 container remove bd0e7dd9056c94da2c9c80bdcd99cf8418369635966a3572fc536e5bb97fa7c5 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Paul Cuzner , build-date=2022-03-28T10:36:18.413762, distribution-scope=public, vcs-type=git, vendor=Red Hat, Inc., version=8.5, release=236.1648460182, io.openshift.expose-services=, name=ubi8, io.openshift.tags=base rhel8, description=Ceph Grafana Container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, io.buildah.version=1.24.2, summary=Grafana Container configured for Ceph mgr/dashboard integration, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, com.redhat.component=ubi8-container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.display-name=Red Hat Universal Base Image 8) 2024-03-22T05:29:38.544 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:29:38 smithi186 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@grafana.a.service: Main process exited, code=exited, status=1/FAILURE 2024-03-22T05:29:38.544 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:29:38 smithi186 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-03-22T05:29:39.044 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:29:38 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:29:38.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.5\", device_class=\"ssd\", hostname=\"smithi005\", instance=\"172.21.15.186:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.5\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"f59f36e2-e80a-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.5\", device_class=\"ssd\", hostname=\"smithi005\", instance=\"smithi186:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.5\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-22T05:29:39.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:29:39 smithi186 ceph-mon[25720]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:29:39.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:29:39 smithi005 ceph-mon[29523]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:29:39.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:29:39 smithi005 ceph-mon[24793]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:29:41.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:29:41 smithi005 ceph-mon[29523]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:29:41.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:29:41 smithi005 ceph-mon[24793]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:29:41.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:29:41 smithi186 ceph-mon[25720]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:29:41.794 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:29:41 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:29:41] "GET /metrics HTTP/1.1" 200 37951 "" "Prometheus/2.33.4" 2024-03-22T05:29:42.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:29:42 smithi005 ceph-mon[29523]: from='client.25054 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:29:42.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:29:42 smithi005 ceph-mon[24793]: from='client.25054 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:29:43.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:29:42 smithi186 ceph-mon[25720]: from='client.25054 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:29:43.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:29:43 smithi186 ceph-mon[25720]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:29:43.794 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:29:43 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:29:43.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi005\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi005\", job=\"node\", machine=\"x86_64\", nodename=\"smithi005\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"f59f36e2-e80a-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi005\", job=\"node\", machine=\"x86_64\", nodename=\"smithi005\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-22T05:29:43.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:29:43 smithi005 ceph-mon[29523]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:29:43.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:29:43 smithi005 ceph-mon[24793]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:29:45.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:29:45 smithi005 ceph-mon[24793]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:29:45.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:29:45 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:29:45.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:29:45 smithi005 ceph-mon[29523]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:29:45.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:29:45 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:29:46.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:29:45 smithi186 ceph-mon[25720]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:29:46.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:29:45 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:29:47.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:29:47 smithi005 ceph-mon[29523]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:29:47.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:29:47 smithi005 ceph-mon[24793]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:29:48.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:29:47 smithi186 ceph-mon[25720]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:29:48.658 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:29:48 smithi186 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@grafana.a.service: Scheduled restart job, restart counter is at 5. 2024-03-22T05:29:48.658 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:29:48 smithi186 systemd[1]: Stopped Ceph grafana.a for f59f36e2-e80a-11ee-95cd-87774f69a715. 2024-03-22T05:29:48.659 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:29:48 smithi186 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@grafana.a.service: Start request repeated too quickly. 2024-03-22T05:29:48.659 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:29:48 smithi186 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-03-22T05:29:48.659 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:29:48 smithi186 systemd[1]: Failed to start Ceph grafana.a for f59f36e2-e80a-11ee-95cd-87774f69a715. 2024-03-22T05:29:49.043 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:29:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:29:48.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.5\", device_class=\"ssd\", hostname=\"smithi005\", instance=\"172.21.15.186:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.5\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"f59f36e2-e80a-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.5\", device_class=\"ssd\", hostname=\"smithi005\", instance=\"smithi186:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.5\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-22T05:29:49.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:29:49 smithi186 ceph-mon[25720]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:29:49.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:29:49 smithi005 ceph-mon[29523]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:29:49.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:29:49 smithi005 ceph-mon[24793]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:29:51.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:29:51 smithi005 ceph-mon[29523]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:29:51.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:29:51 smithi005 ceph-mon[24793]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:29:51.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:29:51 smithi186 ceph-mon[25720]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:29:51.794 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:29:51 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:29:51] "GET /metrics HTTP/1.1" 200 37952 "" "Prometheus/2.33.4" 2024-03-22T05:29:52.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:29:52 smithi005 ceph-mon[29523]: from='client.25054 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:29:52.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:29:52 smithi005 ceph-mon[24793]: from='client.25054 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:29:53.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:29:52 smithi186 ceph-mon[25720]: from='client.25054 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:29:53.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:29:53 smithi186 ceph-mon[25720]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:29:53.794 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:29:53 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:29:53.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi005\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi005\", job=\"node\", machine=\"x86_64\", nodename=\"smithi005\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"f59f36e2-e80a-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi005\", job=\"node\", machine=\"x86_64\", nodename=\"smithi005\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-22T05:29:53.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:29:53 smithi005 ceph-mon[24793]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:29:53.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:29:53 smithi005 ceph-mon[29523]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:29:55.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:29:55 smithi005 ceph-mon[29523]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:29:55.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:29:55 smithi005 ceph-mon[24793]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:29:56.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:29:55 smithi186 ceph-mon[25720]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:29:57.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:29:57 smithi005 ceph-mon[24793]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:29:57.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:29:57 smithi005 ceph-mon[29523]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:29:58.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:29:57 smithi186 ceph-mon[25720]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:29:59.044 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:29:58 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:29:58.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.5\", device_class=\"ssd\", hostname=\"smithi005\", instance=\"172.21.15.186:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.5\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"f59f36e2-e80a-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.5\", device_class=\"ssd\", hostname=\"smithi005\", instance=\"smithi186:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.5\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-22T05:29:59.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:29:59 smithi186 ceph-mon[25720]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:29:59.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:29:59 smithi005 ceph-mon[29523]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:29:59.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:29:59 smithi005 ceph-mon[24793]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:30:00.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:30:00 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:30:00.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:30:00 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:30:00.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:30:00 smithi186 ceph-mon[25720]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-22T05:30:00.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:30:00 smithi186 ceph-mon[25720]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-22T05:30:00.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:30:00 smithi186 ceph-mon[25720]: daemon grafana.a on smithi186 is in unknown state 2024-03-22T05:30:00.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:30:00 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:30:00.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:30:00 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:30:00.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:30:00 smithi005 ceph-mon[24793]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-22T05:30:00.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:30:00 smithi005 ceph-mon[24793]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-22T05:30:00.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:30:00 smithi005 ceph-mon[24793]: daemon grafana.a on smithi186 is in unknown state 2024-03-22T05:30:00.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:30:00 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:30:00.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:30:00 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:30:00.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:30:00 smithi005 ceph-mon[29523]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-22T05:30:00.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:30:00 smithi005 ceph-mon[29523]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-22T05:30:00.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:30:00 smithi005 ceph-mon[29523]: daemon grafana.a on smithi186 is in unknown state 2024-03-22T05:30:01.512 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:30:01 smithi186 ceph-mon[25720]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:30:01.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:30:01 smithi005 ceph-mon[29523]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:30:01.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:30:01 smithi005 ceph-mon[24793]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:30:01.793 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:30:01 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:30:01] "GET /metrics HTTP/1.1" 200 37954 "" "Prometheus/2.33.4" 2024-03-22T05:30:02.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:30:02 smithi186 ceph-mon[25720]: from='client.25054 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:30:02.608 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:30:02 smithi005 ceph-mon[24793]: from='client.25054 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:30:02.609 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:30:02 smithi005 ceph-mon[29523]: from='client.25054 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:30:03.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:30:03 smithi186 ceph-mon[25720]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:30:03.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:30:03 smithi005 ceph-mon[24793]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:30:03.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:30:03 smithi005 ceph-mon[29523]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:30:03.793 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:30:03 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:30:03.503Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi005\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi005\", job=\"node\", machine=\"x86_64\", nodename=\"smithi005\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"f59f36e2-e80a-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi005\", job=\"node\", machine=\"x86_64\", nodename=\"smithi005\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-22T05:30:04.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:30:04 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:30:04.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:30:04 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:30:04.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:30:04 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:30:04.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:30:04 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:30:04.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:30:04 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:30:04.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:30:04 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:30:04.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:30:04 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:30:04.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:30:04 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:30:04.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:30:04 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:30:04.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:30:04 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:30:04.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:30:04 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:30:04.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:30:04 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:30:04.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:30:04 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:30:04.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:30:04 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:30:04.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:30:04 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:30:04.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:30:04 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:30:04.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:30:04 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:30:04.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:30:04 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:30:04.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:30:04 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:30:04.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:30:04 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:30:05.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:30:04 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:30:05.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:30:04 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:30:05.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:30:04 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:30:05.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:30:04 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:30:05.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:30:04 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:30:05.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:30:04 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:30:05.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:30:04 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:30:05.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:30:04 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:30:05.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:30:04 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:30:05.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:30:04 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:30:05.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:30:05 smithi005 ceph-mon[24793]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 857 B/s rd, 0 op/s 2024-03-22T05:30:05.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:30:05 smithi005 ceph-mon[24793]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:30:05.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:30:05 smithi005 ceph-mon[24793]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:30:05.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:30:05 smithi005 ceph-mon[29523]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 857 B/s rd, 0 op/s 2024-03-22T05:30:05.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:30:05 smithi005 ceph-mon[29523]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:30:05.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:30:05 smithi005 ceph-mon[29523]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:30:06.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:30:05 smithi186 ceph-mon[25720]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 857 B/s rd, 0 op/s 2024-03-22T05:30:06.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:30:05 smithi186 ceph-mon[25720]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:30:06.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:30:05 smithi186 ceph-mon[25720]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:30:07.898 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:30:07 smithi005 ceph-mon[24793]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-22T05:30:07.898 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:30:07 smithi005 ceph-mon[29523]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-22T05:30:07.974 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:30:07 smithi186 ceph-mon[25720]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-22T05:30:08.294 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:30:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: 2024-03-22T05:30:07.974+0000 7f0393508700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:30:08.294 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:30:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: Creating ceph-iscsi config... 2024-03-22T05:30:08.294 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:30:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:30:08.294 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:30:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:30:08.295 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:30:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:30:08.295 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:30:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:30:08.295 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:30:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:30:08.295 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:30:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: Traceback (most recent call last): 2024-03-22T05:30:08.295 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:30:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:30:08.295 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:30:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: return _run_code(code, main_globals, None, 2024-03-22T05:30:08.295 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:30:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:30:08.295 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:30:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: exec(code, run_globals) 2024-03-22T05:30:08.295 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:30:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:30:08.295 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:30:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:30:08.295 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:30:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:30:08.295 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:30:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:30:08.295 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:30:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:30:08.295 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:30:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:30:08.295 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:30:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:30:08.296 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:30:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:30:08.296 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:30:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:30:08.296 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:30:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: Traceback (most recent call last): 2024-03-22T05:30:08.296 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:30:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:30:08.296 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:30:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:30:08.296 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:30:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:30:08.296 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:30:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:30:08.296 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:30:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:30:08.296 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:30:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:30:08.296 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:30:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:30:08.296 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:30:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: return future.result(timeout) 2024-03-22T05:30:08.296 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:30:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:30:08.296 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:30:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: return self.__get_result() 2024-03-22T05:30:08.296 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:30:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:30:08.297 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:30:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: raise self._exception 2024-03-22T05:30:08.297 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:30:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:30:08.297 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:30:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: config_blobs=daemon_spec.final_config, 2024-03-22T05:30:08.297 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:30:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:30:08.297 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:30:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:30:08.297 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:30:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:30:08.297 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:30:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: Creating ceph-iscsi config... 2024-03-22T05:30:08.297 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:30:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:30:08.297 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:30:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:30:08.297 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:30:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:30:08.297 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:30:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:30:08.298 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:30:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:30:08.298 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:30:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: Traceback (most recent call last): 2024-03-22T05:30:08.298 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:30:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:30:08.298 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:30:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: return _run_code(code, main_globals, None, 2024-03-22T05:30:08.298 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:30:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:30:08.298 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:30:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: exec(code, run_globals) 2024-03-22T05:30:08.298 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:30:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:30:08.298 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:30:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:30:08.298 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:30:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:30:08.298 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:30:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:30:08.298 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:30:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:30:08.298 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:30:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:30:08.298 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:30:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:30:08.299 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:30:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:30:08.299 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:30:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:30:09.043 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:30:08 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:30:08.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.5\", device_class=\"ssd\", hostname=\"smithi005\", instance=\"172.21.15.186:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.5\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"f59f36e2-e80a-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.5\", device_class=\"ssd\", hostname=\"smithi005\", instance=\"smithi186:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.5\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-22T05:30:09.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:30:09 smithi186 ceph-mon[25720]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:30:09.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:30:09.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:30:09.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:30:09.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:30:09.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:30:09.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:30:09.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:30:09.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:30:09.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:30:09.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:30:09.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:30:09.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:30:09.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:30:09.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:30:09.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:30:09.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:30:09.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:30:09.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:30:09.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:30:09.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:30:09.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:30:09.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:30:09.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:30:09.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:30:09.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:30:09.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:30:09.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:30:09.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:30:09.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: return future.result(timeout) 2024-03-22T05:30:09.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:30:09.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.__get_result() 2024-03-22T05:30:09.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:30:09.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: raise self._exception 2024-03-22T05:30:09.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:30:09.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:30:09.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:30:09.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:30:09.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:30:09.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:30:09.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:30:09.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:30:09.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:30:09.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:30:09.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:30:09.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:30:09.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:30:09.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:30:09.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:30:09.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:30:09.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:30:09.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:30:09.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:30:09.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:30:09.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:30:09.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:30:09.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:30:09.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:30:09.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:30:09.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:30:09 smithi186 ceph-mon[25720]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 857 B/s rd, 0 op/s 2024-03-22T05:30:09.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:30:09 smithi005 ceph-mon[24793]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:30:09.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:30:09.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:30:09.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:30:09.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:30:09.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:30:09.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:30:09.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:30:09.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:30:09.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:30:09.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:30:09.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:30:09.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:30:09.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:30:09.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:30:09.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:30:09.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:30:09.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:30:09.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:30:09.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:30:09.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:30:09.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:30:09.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:30:09.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:30:09.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:30:09.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:30:09.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:30:09.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:30:09.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:30:09.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: return future.result(timeout) 2024-03-22T05:30:09.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:30:09.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.__get_result() 2024-03-22T05:30:09.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:30:09.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: raise self._exception 2024-03-22T05:30:09.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:30:09.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:30:09.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:30:09.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:30:09.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:30:09.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:30:09.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:30:09.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:30:09.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:30:09.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:30:09.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:30:09.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:30:09.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:30:09.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:30:09.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:30:09.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:30:09.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:30:09.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:30:09.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:30:09.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:30:09.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:30:09.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:30:09.707 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:30:09.707 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:30:09.707 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:30:09.707 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:30:09 smithi005 ceph-mon[24793]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 857 B/s rd, 0 op/s 2024-03-22T05:30:09.707 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:30:09 smithi005 ceph-mon[29523]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:30:09.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:30:09.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:30:09.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:30:09.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:30:09.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:30:09.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:30:09.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:30:09.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:30:09.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:30:09.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:30:09.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:30:09.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:30:09.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:30:09.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:30:09.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:30:09.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:30:09.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:30:09.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:30:09.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:30:09.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:30:09.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:30:09.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:30:09.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:30:09.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:30:09.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:30:09.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:30:09.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:30:09.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:30:09.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: return future.result(timeout) 2024-03-22T05:30:09.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:30:09.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.__get_result() 2024-03-22T05:30:09.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:30:09.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: raise self._exception 2024-03-22T05:30:09.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:30:09.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:30:09.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:30:09.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:30:09.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:30:09.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:30:09.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:30:09.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:30:09.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:30:09.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:30:09.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:30:09.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:30:09.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:30:09.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:30:09.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:30:09.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:30:09.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:30:09.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:30:09.711 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:30:09.711 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:30:09.711 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:30:09.711 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:30:09.711 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:30:09.711 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:30:09.711 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:30:09.711 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:30:09 smithi005 ceph-mon[29523]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 857 B/s rd, 0 op/s 2024-03-22T05:30:11.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:30:11 smithi186 ceph-mon[25720]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 857 B/s rd, 0 op/s 2024-03-22T05:30:11.794 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:30:11 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:30:11] "GET /metrics HTTP/1.1" 200 37954 "" "Prometheus/2.33.4" 2024-03-22T05:30:11.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:30:11 smithi005 ceph-mon[24793]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 857 B/s rd, 0 op/s 2024-03-22T05:30:11.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:30:11 smithi005 ceph-mon[29523]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 857 B/s rd, 0 op/s 2024-03-22T05:30:13.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:30:13 smithi186 ceph-mon[25720]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-22T05:30:13.794 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:30:13 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:30:13.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi005\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi005\", job=\"node\", machine=\"x86_64\", nodename=\"smithi005\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"f59f36e2-e80a-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi005\", job=\"node\", machine=\"x86_64\", nodename=\"smithi005\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-22T05:30:13.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:30:13 smithi005 ceph-mon[29523]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-22T05:30:13.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:30:13 smithi005 ceph-mon[24793]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-22T05:30:15.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:30:15 smithi186 ceph-mon[25720]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 600 B/s rd, 0 op/s 2024-03-22T05:30:15.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:30:15 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:30:15.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:30:15 smithi005 ceph-mon[29523]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 600 B/s rd, 0 op/s 2024-03-22T05:30:15.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:30:15 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:30:15.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:30:15 smithi005 ceph-mon[24793]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 600 B/s rd, 0 op/s 2024-03-22T05:30:15.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:30:15 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:30:17.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:30:17 smithi186 ceph-mon[25720]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-03-22T05:30:17.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:30:17 smithi005 ceph-mon[24793]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-03-22T05:30:17.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:30:17 smithi005 ceph-mon[29523]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-03-22T05:30:19.044 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:30:18 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:30:18.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.5\", device_class=\"ssd\", hostname=\"smithi005\", instance=\"172.21.15.186:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.5\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"f59f36e2-e80a-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.5\", device_class=\"ssd\", hostname=\"smithi005\", instance=\"smithi186:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.5\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-22T05:30:19.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:30:19 smithi186 ceph-mon[25720]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-22T05:30:19.596 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:30:19 smithi005 ceph-mon[24793]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-22T05:30:19.596 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:30:19 smithi005 ceph-mon[29523]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-22T05:30:20.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:30:20 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/573962559' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-22T05:30:20.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:30:20 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/573962559' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-22T05:30:20.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:30:20 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/573962559' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-22T05:30:21.511 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:30:21 smithi186 ceph-mon[25720]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-22T05:30:21.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:30:21 smithi005 ceph-mon[24793]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-22T05:30:21.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:30:21 smithi005 ceph-mon[29523]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-22T05:30:21.793 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:30:21 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:30:21] "GET /metrics HTTP/1.1" 200 37963 "" "Prometheus/2.33.4" 2024-03-22T05:30:23.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:30:23 smithi186 ceph-mon[25720]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-22T05:30:23.794 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:30:23 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:30:23.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi005\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi005\", job=\"node\", machine=\"x86_64\", nodename=\"smithi005\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"f59f36e2-e80a-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi005\", job=\"node\", machine=\"x86_64\", nodename=\"smithi005\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-22T05:30:23.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:30:23 smithi005 ceph-mon[29523]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-22T05:30:23.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:30:23 smithi005 ceph-mon[24793]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-22T05:30:25.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:30:25 smithi186 ceph-mon[25720]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-22T05:30:25.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:30:25 smithi005 ceph-mon[29523]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-22T05:30:25.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:30:25 smithi005 ceph-mon[24793]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-22T05:30:27.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:30:27 smithi186 ceph-mon[25720]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-03-22T05:30:27.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:30:27 smithi005 ceph-mon[29523]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-03-22T05:30:27.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:30:27 smithi005 ceph-mon[24793]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-03-22T05:30:29.044 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:30:28 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:30:28.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.5\", device_class=\"ssd\", hostname=\"smithi005\", instance=\"172.21.15.186:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.5\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"f59f36e2-e80a-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.5\", device_class=\"ssd\", hostname=\"smithi005\", instance=\"smithi186:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.5\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-22T05:30:29.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:30:29 smithi186 ceph-mon[25720]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-03-22T05:30:29.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:30:29 smithi005 ceph-mon[29523]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-03-22T05:30:29.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:30:29 smithi005 ceph-mon[24793]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-03-22T05:30:30.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:30:30 smithi186 ceph-mon[25720]: from='client.25081 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:30:30.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:30:30 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:30:30.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:30:30 smithi005 ceph-mon[24793]: from='client.25081 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:30:30.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:30:30 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:30:30.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:30:30 smithi005 ceph-mon[29523]: from='client.25081 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:30:30.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:30:30 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:30:31.511 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:30:31 smithi186 ceph-mon[25720]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-03-22T05:30:31.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:30:31 smithi005 ceph-mon[29523]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-03-22T05:30:31.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:30:31 smithi005 ceph-mon[24793]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-03-22T05:30:31.793 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:30:31 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:30:31] "GET /metrics HTTP/1.1" 200 37962 "" "Prometheus/2.33.4" 2024-03-22T05:30:33.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:30:33 smithi186 ceph-mon[25720]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-03-22T05:30:33.794 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:30:33 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:30:33.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi005\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi005\", job=\"node\", machine=\"x86_64\", nodename=\"smithi005\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"f59f36e2-e80a-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi005\", job=\"node\", machine=\"x86_64\", nodename=\"smithi005\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-22T05:30:33.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:30:33 smithi005 ceph-mon[29523]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-03-22T05:30:33.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:30:33 smithi005 ceph-mon[24793]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-03-22T05:30:35.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:30:35 smithi186 ceph-mon[25720]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:30:35.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:30:35 smithi005 ceph-mon[24793]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:30:35.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:30:35 smithi005 ceph-mon[29523]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:30:37.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:30:37 smithi186 ceph-mon[25720]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:30:37.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:30:37 smithi005 ceph-mon[29523]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:30:37.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:30:37 smithi005 ceph-mon[24793]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:30:39.044 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:30:38 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:30:38.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.5\", device_class=\"ssd\", hostname=\"smithi005\", instance=\"172.21.15.186:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.5\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"f59f36e2-e80a-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.5\", device_class=\"ssd\", hostname=\"smithi005\", instance=\"smithi186:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.5\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-22T05:30:39.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:30:39 smithi186 ceph-mon[25720]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:30:39.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:30:39 smithi005 ceph-mon[29523]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:30:39.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:30:39 smithi005 ceph-mon[24793]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:30:40.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:30:40 smithi186 ceph-mon[25720]: from='client.25081 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:30:40.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:30:40 smithi005 ceph-mon[24793]: from='client.25081 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:30:40.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:30:40 smithi005 ceph-mon[29523]: from='client.25081 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:30:41.511 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:30:41 smithi186 ceph-mon[25720]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:30:41.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:30:41 smithi005 ceph-mon[24793]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:30:41.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:30:41 smithi005 ceph-mon[29523]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:30:41.793 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:30:41 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:30:41] "GET /metrics HTTP/1.1" 200 37962 "" "Prometheus/2.33.4" 2024-03-22T05:30:43.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:30:43 smithi186 ceph-mon[25720]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:30:43.794 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:30:43 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:30:43.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi005\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi005\", job=\"node\", machine=\"x86_64\", nodename=\"smithi005\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"f59f36e2-e80a-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi005\", job=\"node\", machine=\"x86_64\", nodename=\"smithi005\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-22T05:30:43.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:30:43 smithi005 ceph-mon[24793]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:30:43.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:30:43 smithi005 ceph-mon[29523]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:30:45.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:30:45 smithi186 ceph-mon[25720]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:30:45.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:30:45 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:30:45.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:30:45 smithi005 ceph-mon[24793]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:30:45.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:30:45 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:30:45.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:30:45 smithi005 ceph-mon[29523]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:30:45.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:30:45 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:30:47.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:30:47 smithi186 ceph-mon[25720]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:30:47.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:30:47 smithi005 ceph-mon[24793]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:30:47.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:30:47 smithi005 ceph-mon[29523]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:30:49.044 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:30:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:30:48.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.5\", device_class=\"ssd\", hostname=\"smithi005\", instance=\"172.21.15.186:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.5\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"f59f36e2-e80a-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.5\", device_class=\"ssd\", hostname=\"smithi005\", instance=\"smithi186:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.5\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-22T05:30:49.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:30:49 smithi186 ceph-mon[25720]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:30:49.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:30:49 smithi005 ceph-mon[24793]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:30:49.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:30:49 smithi005 ceph-mon[29523]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:30:50.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:30:50 smithi186 ceph-mon[25720]: from='client.25081 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:30:50.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:30:50 smithi005 ceph-mon[24793]: from='client.25081 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:30:50.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:30:50 smithi005 ceph-mon[29523]: from='client.25081 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:30:51.512 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:30:51 smithi186 ceph-mon[25720]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:30:51.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:30:51 smithi005 ceph-mon[24793]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:30:51.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:30:51 smithi005 ceph-mon[29523]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:30:51.793 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:30:51 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:30:51] "GET /metrics HTTP/1.1" 200 37963 "" "Prometheus/2.33.4" 2024-03-22T05:30:53.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:30:53 smithi186 ceph-mon[25720]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:30:53.794 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:30:53 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:30:53.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi005\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi005\", job=\"node\", machine=\"x86_64\", nodename=\"smithi005\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"f59f36e2-e80a-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi005\", job=\"node\", machine=\"x86_64\", nodename=\"smithi005\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-22T05:30:53.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:30:53 smithi005 ceph-mon[24793]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:30:53.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:30:53 smithi005 ceph-mon[29523]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:30:55.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:30:55 smithi186 ceph-mon[25720]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:30:55.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:30:55 smithi005 ceph-mon[29523]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:30:55.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:30:55 smithi005 ceph-mon[24793]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:30:57.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:30:57 smithi186 ceph-mon[25720]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:30:57.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:30:57 smithi005 ceph-mon[24793]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:30:57.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:30:57 smithi005 ceph-mon[29523]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:30:59.043 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:30:58 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:30:58.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.5\", device_class=\"ssd\", hostname=\"smithi005\", instance=\"172.21.15.186:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.5\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"f59f36e2-e80a-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.5\", device_class=\"ssd\", hostname=\"smithi005\", instance=\"smithi186:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.5\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-22T05:30:59.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:30:59 smithi186 ceph-mon[25720]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:30:59.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:30:59 smithi005 ceph-mon[29523]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:30:59.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:30:59 smithi005 ceph-mon[24793]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:31:00.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:00 smithi186 ceph-mon[25720]: from='client.25081 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:31:00.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:00 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:31:00.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:00 smithi005 ceph-mon[29523]: from='client.25081 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:31:00.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:00 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:31:00.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:00 smithi005 ceph-mon[24793]: from='client.25081 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:31:00.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:00 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:31:01.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:01 smithi186 ceph-mon[25720]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:31:01.544 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:31:01 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:31:01] "GET /metrics HTTP/1.1" 200 37962 "" "Prometheus/2.33.4" 2024-03-22T05:31:01.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:01 smithi005 ceph-mon[29523]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:31:01.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:01 smithi005 ceph-mon[24793]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:31:02.925 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph orch daemon redeploy "mgr.$(ceph mgr dump -f json | jq .standbys | jq .[] | jq -r .name)" --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-03-22T05:31:03.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:03 smithi186 ceph-mon[25720]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:31:03.794 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:31:03 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:31:03.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi005\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi005\", job=\"node\", machine=\"x86_64\", nodename=\"smithi005\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"f59f36e2-e80a-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi005\", job=\"node\", machine=\"x86_64\", nodename=\"smithi005\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-22T05:31:03.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:03 smithi005 ceph-mon[29523]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:31:03.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:03 smithi005 ceph-mon[24793]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:31:05.563 INFO:teuthology.orchestra.run.smithi005.stdout:Scheduled to redeploy mgr.y on host 'smithi005' 2024-03-22T05:31:05.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:05 smithi186 ceph-mon[25720]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:31:05.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:05 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/657910901' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-22T05:31:05.829 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:05 smithi005 ceph-mon[29523]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:31:05.829 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:05 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/657910901' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-22T05:31:05.829 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:05 smithi005 ceph-mon[24793]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:31:05.829 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:05 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/657910901' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-22T05:31:06.387 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph orch ps --refresh' 2024-03-22T05:31:06.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:06 smithi005 ceph-mon[24793]: from='client.15186 -' entity='client.admin' cmd=[{"prefix": "orch daemon redeploy", "name": "mgr.y", "image": "quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:31:06.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:06 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:31:06.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:06 smithi005 ceph-mon[24793]: Schedule redeploy daemon mgr.y 2024-03-22T05:31:06.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:06 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:31:06.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:06 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:31:06.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:06 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:31:06.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:06 smithi005 ceph-mon[29523]: from='client.15186 -' entity='client.admin' cmd=[{"prefix": "orch daemon redeploy", "name": "mgr.y", "image": "quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:31:06.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:06 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:31:06.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:06 smithi005 ceph-mon[29523]: Schedule redeploy daemon mgr.y 2024-03-22T05:31:06.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:06 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:31:06.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:06 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:31:06.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:06 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:31:07.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:06 smithi186 ceph-mon[25720]: from='client.15186 -' entity='client.admin' cmd=[{"prefix": "orch daemon redeploy", "name": "mgr.y", "image": "quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:31:07.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:06 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:31:07.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:06 smithi186 ceph-mon[25720]: Schedule redeploy daemon mgr.y 2024-03-22T05:31:07.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:06 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:31:07.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:06 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:31:07.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:06 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:31:07.853 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:07 smithi005 ceph-mon[24793]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:31:07.853 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:07 smithi005 ceph-mon[29523]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:31:08.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:07 smithi186 ceph-mon[25720]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:31:08.782 INFO:teuthology.orchestra.run.smithi005.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T05:31:08.782 INFO:teuthology.orchestra.run.smithi005.stdout:alertmanager.a smithi005 *:9093,9094 running (2m) 64s ago 10m 14.6M - 0.23.0 ba2b418f427c fd7dc5dbe3e5 2024-03-22T05:31:08.782 INFO:teuthology.orchestra.run.smithi005.stdout:grafana.a smithi186 *:3000 error 65s ago 10m - - 2024-03-22T05:31:08.782 INFO:teuthology.orchestra.run.smithi005.stdout:iscsi.foo.smithi005.mcdwbp smithi005 running (118s) 64s ago 9m 48.3M - 3.5 e1d6a67b021e 3ff56657eeab 2024-03-22T05:31:08.782 INFO:teuthology.orchestra.run.smithi005.stdout:mgr.x smithi186 *:8443,9283 running (5m) 65s ago 15m 510M - 19.0.0-2299-gea01217f aa461e7c5e7f 66924e151e7b 2024-03-22T05:31:08.782 INFO:teuthology.orchestra.run.smithi005.stdout:mgr.y smithi005 *:9283 running (16m) 64s ago 16m 411M - 17.2.0 e1d6a67b021e fcee4f81e3ee 2024-03-22T05:31:08.783 INFO:teuthology.orchestra.run.smithi005.stdout:mon.a smithi005 running (16m) 64s ago 16m 55.7M 2048M 17.2.0 e1d6a67b021e 38e0bba4bb03 2024-03-22T05:31:08.783 INFO:teuthology.orchestra.run.smithi005.stdout:mon.b smithi186 running (15m) 65s ago 15m 44.6M 2048M 17.2.0 e1d6a67b021e 323f53f932f6 2024-03-22T05:31:08.783 INFO:teuthology.orchestra.run.smithi005.stdout:mon.c smithi005 running (15m) 64s ago 15m 43.7M 2048M 17.2.0 e1d6a67b021e 00ec24d38612 2024-03-22T05:31:08.783 INFO:teuthology.orchestra.run.smithi005.stdout:node-exporter.a smithi005 *:9100 running (2m) 64s ago 11m 13.5M - 1.3.1 1dbe0e931976 a9085d6042eb 2024-03-22T05:31:08.783 INFO:teuthology.orchestra.run.smithi005.stdout:node-exporter.b smithi186 *:9100 running (2m) 65s ago 11m 9.87M - 1.3.1 1dbe0e931976 74aa1181ff71 2024-03-22T05:31:08.783 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0 smithi005 running (14m) 64s ago 14m 51.4M 2993M 17.2.0 e1d6a67b021e ef5c7053dd37 2024-03-22T05:31:08.783 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1 smithi005 running (14m) 64s ago 14m 53.6M 2993M 17.2.0 e1d6a67b021e a1e7ee1017b6 2024-03-22T05:31:08.783 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2 smithi005 running (13m) 64s ago 13m 47.0M 2993M 17.2.0 e1d6a67b021e 462ea63ad0e5 2024-03-22T05:31:08.783 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3 smithi005 running (13m) 64s ago 13m 53.6M 2993M 17.2.0 e1d6a67b021e b25432a7239b 2024-03-22T05:31:08.783 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4 smithi186 running (13m) 65s ago 13m 52.3M 3248M 17.2.0 e1d6a67b021e cd963671f340 2024-03-22T05:31:08.783 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5 smithi186 running (12m) 65s ago 12m 49.2M 3248M 17.2.0 e1d6a67b021e f98c87f1a2cd 2024-03-22T05:31:08.783 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6 smithi186 running (12m) 65s ago 12m 47.9M 3248M 17.2.0 e1d6a67b021e 704ac49c68c4 2024-03-22T05:31:08.783 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7 smithi186 running (12m) 65s ago 12m 50.5M 3248M 17.2.0 e1d6a67b021e f415b90edf63 2024-03-22T05:31:08.783 INFO:teuthology.orchestra.run.smithi005.stdout:prometheus.a smithi186 *:9095 running (2m) 65s ago 11m 64.2M - 2.33.4 514e6a882f6e c3a69c1f7e79 2024-03-22T05:31:08.784 INFO:teuthology.orchestra.run.smithi005.stdout:rgw.foo.smithi005.exdhqu smithi005 *:8000 running (9m) 64s ago 9m 74.9M - 17.2.0 e1d6a67b021e 6bb27fe11553 2024-03-22T05:31:08.784 INFO:teuthology.orchestra.run.smithi005.stdout:rgw.foo.smithi186.mcgodk smithi186 *:8000 running (9m) 65s ago 9m 72.9M - 17.2.0 e1d6a67b021e 19b5c20a34f7 2024-03-22T05:31:09.044 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:31:08 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:31:08.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.5\", device_class=\"ssd\", hostname=\"smithi005\", instance=\"172.21.15.186:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.5\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"f59f36e2-e80a-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.5\", device_class=\"ssd\", hostname=\"smithi005\", instance=\"smithi186:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.5\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-22T05:31:09.529 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:09 smithi005 ceph-mon[24793]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:31:09.529 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:09 smithi005 ceph-mon[24793]: from='client.25105 -' entity='client.admin' cmd=[{"prefix": "orch ps", "refresh": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:31:09.530 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:09 smithi005 ceph-mon[29523]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:31:09.530 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:09 smithi005 ceph-mon[29523]: from='client.25105 -' entity='client.admin' cmd=[{"prefix": "orch ps", "refresh": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:31:09.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:09 smithi186 ceph-mon[25720]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:31:09.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:09 smithi186 ceph-mon[25720]: from='client.25105 -' entity='client.admin' cmd=[{"prefix": "orch ps", "refresh": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:31:09.551 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'sleep 180' 2024-03-22T05:31:10.434 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:10 smithi005 ceph-mon[24793]: from='client.25081 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:31:10.435 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:10 smithi005 ceph-mon[29523]: from='client.25081 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:31:10.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:10 smithi186 ceph-mon[25720]: from='client.25081 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:31:11.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:11 smithi186 ceph-mon[25720]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:31:11.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:11 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:31:11.794 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:31:11 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:31:11] "GET /metrics HTTP/1.1" 200 37962 "" "Prometheus/2.33.4" 2024-03-22T05:31:11.845 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:11 smithi005 ceph-mon[24793]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:31:11.845 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:11 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:31:11.845 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:11 smithi005 ceph-mon[29523]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:31:11.845 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:11 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:31:13.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:13 smithi186 ceph-mon[25720]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:31:13.794 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:31:13 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:31:13.503Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi005\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi005\", job=\"node\", machine=\"x86_64\", nodename=\"smithi005\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"f59f36e2-e80a-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi005\", job=\"node\", machine=\"x86_64\", nodename=\"smithi005\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-22T05:31:13.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:13 smithi005 ceph-mon[24793]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:31:13.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:13 smithi005 ceph-mon[29523]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:31:15.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:15 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:31:15.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:15 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:31:15.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:15 smithi186 ceph-mon[25720]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:31:15.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:15 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:31:15.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:15 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:31:15.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:15 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:31:15.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:15 smithi005 ceph-mon[24793]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:31:15.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:15 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:31:15.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:15 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:31:15.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:15 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:31:15.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:15 smithi005 ceph-mon[29523]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:31:15.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:15 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:31:16.667 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:16 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:31:16.667 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:16 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:31:16.667 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:16 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:31:16.667 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:16 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:31:16.667 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:16 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:31:16.667 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:16 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-22T05:31:16.667 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:16 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-22T05:31:16.667 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:16 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-22T05:31:16.667 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:16 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:31:16.668 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:16 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:31:16.668 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:16 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:31:16.668 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:16 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:31:16.668 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:16 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:31:16.668 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:16 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:31:16.668 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:16 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-22T05:31:16.668 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:16 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-22T05:31:16.668 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:16 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-22T05:31:16.668 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:16 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:31:16.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:16 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:31:16.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:16 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:31:16.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:16 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:31:16.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:16 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:31:16.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:16 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:31:16.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:16 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-22T05:31:16.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:16 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-22T05:31:16.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:16 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-22T05:31:16.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:16 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:31:17.670 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:17 smithi005 ceph-mon[24793]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 63 op/s 2024-03-22T05:31:17.670 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:17 smithi005 ceph-mon[24793]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 0 B/s wr, 75 op/s 2024-03-22T05:31:17.671 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:17 smithi005 ceph-mon[24793]: Deploying daemon mgr.y on smithi005 2024-03-22T05:31:17.671 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:17 smithi005 ceph-mon[29523]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 63 op/s 2024-03-22T05:31:17.671 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:17 smithi005 ceph-mon[29523]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 0 B/s wr, 75 op/s 2024-03-22T05:31:17.671 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:17 smithi005 ceph-mon[29523]: Deploying daemon mgr.y on smithi005 2024-03-22T05:31:17.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:17 smithi186 ceph-mon[25720]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 63 op/s 2024-03-22T05:31:17.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:17 smithi186 ceph-mon[25720]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 0 B/s wr, 75 op/s 2024-03-22T05:31:17.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:17 smithi186 ceph-mon[25720]: Deploying daemon mgr.y on smithi005 2024-03-22T05:31:18.251 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:31:17 smithi005 systemd[1]: Stopping Ceph mgr.y for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:31:18.536 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:31:18 smithi005 podman[63067]: 2024-03-22 05:31:18.251105355 +0000 UTC m=+0.142890262 container died fcee4f81e3eeaaaed6b6a2392a8a8419457fcee212e047709c4902351ae4786d (image=quay.io/ceph/ceph:v17.2.0, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y, io.openshift.tags=base centos centos-stream, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., release=754, architecture=x86_64, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, RELEASE=HEAD, maintainer=Guillaume Abrioux , io.openshift.expose-services=, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-03-22T05:31:18.870 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:31:18 smithi005 podman[63067]: 2024-03-22 05:31:18.603383486 +0000 UTC m=+0.495168393 container cleanup fcee4f81e3eeaaaed6b6a2392a8a8419457fcee212e047709c4902351ae4786d (image=quay.io/ceph/ceph:v17.2.0, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, vcs-type=git, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, io.buildah.version=1.19.8, version=8, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, ceph=True, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8) 2024-03-22T05:31:18.870 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:31:18 smithi005 bash[63067]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y 2024-03-22T05:31:19.043 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:31:18 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:31:18.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.5\", device_class=\"ssd\", hostname=\"smithi005\", instance=\"172.21.15.186:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.5\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"f59f36e2-e80a-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.5\", device_class=\"ssd\", hostname=\"smithi005\", instance=\"smithi186:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.5\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-22T05:31:19.202 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:31:18 smithi005 podman[63081]: 2024-03-22 05:31:18.871290385 +0000 UTC m=+0.624906109 container remove fcee4f81e3eeaaaed6b6a2392a8a8419457fcee212e047709c4902351ae4786d (image=quay.io/ceph/ceph:v17.2.0, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, distribution-scope=public, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, ceph=True, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-03-22T05:31:19.202 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:31:18 smithi005 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-03-22T05:31:19.202 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:31:19 smithi005 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@mgr.y.service: Failed with result 'exit-code'. 2024-03-22T05:31:19.202 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:31:19 smithi005 systemd[1]: Stopped Ceph mgr.y for f59f36e2-e80a-11ee-95cd-87774f69a715. 2024-03-22T05:31:19.202 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:31:19 smithi005 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@mgr.y.service: Consumed 1min 2.159s CPU time. 2024-03-22T05:31:19.488 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:19 smithi005 ceph-mon[24793]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 0 B/s wr, 75 op/s 2024-03-22T05:31:19.488 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:19 smithi005 ceph-mon[29523]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 0 B/s wr, 75 op/s 2024-03-22T05:31:19.488 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:31:19 smithi005 systemd[1]: Starting Ceph mgr.y for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:31:19.488 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:31:19 smithi005 podman[63215]: 2024-03-22 05:31:19.367962948 +0000 UTC m=+0.020386382 image pull aa461e7c5e7f322b169c789b389d594b994ea7ffcbc9e24e563a3ec9b69bc3c1 quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9 2024-03-22T05:31:19.743 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:31:19 smithi005 podman[63215]: 2024-03-22 05:31:19.488886269 +0000 UTC m=+0.141309696 container create 62a813eebc0db2b81d3553d8158695697ee8f9dd162b89616f3e4e05a385cffb (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-03-22T05:31:19.743 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:31:19 smithi005 podman[63215]: 2024-03-22 05:31:19.716479526 +0000 UTC m=+0.368902968 container init 62a813eebc0db2b81d3553d8158695697ee8f9dd162b89616f3e4e05a385cffb (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-22T05:31:19.743 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:31:19 smithi005 podman[63215]: 2024-03-22 05:31:19.719021833 +0000 UTC m=+0.371445262 container start 62a813eebc0db2b81d3553d8158695697ee8f9dd162b89616f3e4e05a385cffb (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-22T05:31:19.743 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:31:19 smithi005 ceph-mgr[63233]: -- 172.21.15.5:0/1917162138 <== mon.0 v2:172.21.15.5:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e51b0405a0 con 0x55e51b014c00 2024-03-22T05:31:19.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:19 smithi186 ceph-mon[25720]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 0 B/s wr, 75 op/s 2024-03-22T05:31:20.202 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:31:19 smithi005 bash[63215]: 62a813eebc0db2b81d3553d8158695697ee8f9dd162b89616f3e4e05a385cffb 2024-03-22T05:31:20.202 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:31:19 smithi005 systemd[1]: Started Ceph mgr.y for f59f36e2-e80a-11ee-95cd-87774f69a715. 2024-03-22T05:31:20.644 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:31:20 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:31:20.295+0000 7f84a310d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-22T05:31:20.644 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:31:20 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:31:20.399+0000 7f84a310d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-22T05:31:20.644 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:20 smithi005 ceph-mon[24793]: from='client.25081 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:31:20.644 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:20 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:31:20.645 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:20 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:31:20.645 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:20 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:31:20.645 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:20 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:31:20.645 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:20 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:31:20.645 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:20 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:31:20.645 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:20 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:31:20.645 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:20 smithi005 ceph-mon[29523]: from='client.25081 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:31:20.645 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:20 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:31:20.645 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:20 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:31:20.645 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:20 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:31:20.645 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:20 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:31:20.645 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:20 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:31:20.646 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:20 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:31:20.646 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:20 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:31:20.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:20 smithi186 ceph-mon[25720]: from='client.25081 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:31:20.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:20 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:31:20.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:20 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:31:20.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:20 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:31:20.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:20 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:31:20.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:20 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:31:20.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:20 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:31:20.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:20 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:31:21.401 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:31:21 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:31:21.106+0000 7f84a310d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-22T05:31:21.402 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:31:21 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:31:21.201+0000 7f84a310d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-22T05:31:21.402 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:31:21 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:31:21.346+0000 7f84a310d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-22T05:31:21.702 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:31:21 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:31:21.446+0000 7f84a310d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-22T05:31:21.702 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:31:21 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:31:21.535+0000 7f84a310d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-22T05:31:21.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:21 smithi005 ceph-mon[24793]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:31:21.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:21 smithi005 ceph-mon[24793]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:31:21.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:21 smithi005 ceph-mon[24793]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 0 B/s wr, 75 op/s 2024-03-22T05:31:21.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:21 smithi005 ceph-mon[29523]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:31:21.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:21 smithi005 ceph-mon[29523]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:31:21.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:21 smithi005 ceph-mon[29523]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 0 B/s wr, 75 op/s 2024-03-22T05:31:21.793 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:31:21 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:31:21] "GET /metrics HTTP/1.1" 200 37962 "" "Prometheus/2.33.4" 2024-03-22T05:31:21.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:21 smithi186 ceph-mon[25720]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:31:21.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:21 smithi186 ceph-mon[25720]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:31:21.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:21 smithi186 ceph-mon[25720]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 0 B/s wr, 75 op/s 2024-03-22T05:31:22.122 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:31:21 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:31:21.815+0000 7f84a310d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-22T05:31:22.122 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:31:21 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:31:21.930+0000 7f84a310d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-22T05:31:22.123 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:31:22 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:31:22.021+0000 7f84a310d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-22T05:31:22.123 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:31:22 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:31:22.122+0000 7f84a310d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-22T05:31:22.407 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:31:22 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:31:22.305+0000 7f84a310d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-22T05:31:23.294 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:31:23 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: 2024-03-22T05:31:22.999+0000 7f0393508700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:31:23.294 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:31:23 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: Creating ceph-iscsi config... 2024-03-22T05:31:23.295 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:31:23 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:31:23.295 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:31:23 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:31:23.295 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:31:23 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:31:23.295 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:31:23 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:31:23.295 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:31:23 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:31:23.295 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:31:23 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: Traceback (most recent call last): 2024-03-22T05:31:23.295 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:31:23 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:31:23.295 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:31:23 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: return _run_code(code, main_globals, None, 2024-03-22T05:31:23.295 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:31:23 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:31:23.295 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:31:23 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: exec(code, run_globals) 2024-03-22T05:31:23.295 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:31:23 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:31:23.295 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:31:23 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:31:23.295 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:31:23 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:31:23.295 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:31:23 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:31:23.296 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:31:23 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:31:23.296 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:31:23 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:31:23.296 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:31:23 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:31:23.296 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:31:23 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:31:23.296 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:31:23 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:31:23.296 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:31:23 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: Traceback (most recent call last): 2024-03-22T05:31:23.296 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:31:23 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:31:23.297 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:31:23 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:31:23.297 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:31:23 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:31:23.297 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:31:23 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:31:23.297 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:31:23 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:31:23.297 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:31:23 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:31:23.297 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:31:23 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:31:23.297 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:31:23 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: return future.result(timeout) 2024-03-22T05:31:23.297 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:31:23 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:31:23.297 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:31:23 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: return self.__get_result() 2024-03-22T05:31:23.297 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:31:23 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:31:23.297 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:31:23 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: raise self._exception 2024-03-22T05:31:23.297 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:31:23 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:31:23.297 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:31:23 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: config_blobs=daemon_spec.final_config, 2024-03-22T05:31:23.297 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:31:23 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:31:23.297 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:31:23 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:31:23.298 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:31:23 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:31:23.298 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:31:23 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: Creating ceph-iscsi config... 2024-03-22T05:31:23.298 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:31:23 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:31:23.298 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:31:23 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:31:23.298 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:31:23 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:31:23.298 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:31:23 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:31:23.298 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:31:23 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:31:23.298 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:31:23 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: Traceback (most recent call last): 2024-03-22T05:31:23.298 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:31:23 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:31:23.298 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:31:23 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: return _run_code(code, main_globals, None, 2024-03-22T05:31:23.298 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:31:23 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:31:23.298 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:31:23 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: exec(code, run_globals) 2024-03-22T05:31:23.298 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:31:23 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:31:23.299 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:31:23 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:31:23.299 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:31:23 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:31:23.299 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:31:23 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:31:23.299 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:31:23 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:31:23.299 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:31:23 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:31:23.299 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:31:23 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:31:23.299 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:31:23 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:31:23.299 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:31:23 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:31:23.505 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:23 smithi186 ceph-mon[25720]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 91 KiB/s rd, 0 B/s wr, 150 op/s 2024-03-22T05:31:23.505 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:23 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:31:23.505 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:31:23 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:31:23.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi005\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi005\", job=\"node\", machine=\"x86_64\", nodename=\"smithi005\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"f59f36e2-e80a-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi005\", job=\"node\", machine=\"x86_64\", nodename=\"smithi005\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-22T05:31:23.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:23 smithi005 ceph-mon[24793]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 91 KiB/s rd, 0 B/s wr, 150 op/s 2024-03-22T05:31:23.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:23 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:31:23.703 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:31:23 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:31:23.347+0000 7f84a310d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-22T05:31:23.703 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:31:23 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:31:23.492+0000 7f84a310d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-22T05:31:23.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:23 smithi005 ceph-mon[29523]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 91 KiB/s rd, 0 B/s wr, 150 op/s 2024-03-22T05:31:23.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:23 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:31:24.167 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:31:23 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:31:23.859+0000 7f84a310d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-22T05:31:24.167 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:31:23 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:31:23.957+0000 7f84a310d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-22T05:31:24.167 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:31:24 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:31:24.166+0000 7f84a310d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-22T05:31:24.452 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:31:24 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:31:24.361+0000 7f84a310d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-22T05:31:24.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:24 smithi005 ceph-mon[24793]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:31:24.452 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:31:24.452 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:31:24.452 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:31:24.452 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:31:24.453 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:31:24.453 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:31:24.453 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:31:24.453 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:31:24.453 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:31:24.453 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:31:24.453 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:31:24.453 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:31:24.453 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:31:24.453 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:31:24.453 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:31:24.453 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:31:24.453 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:31:24.453 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:31:24.453 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:31:24.454 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:31:24.454 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:31:24.454 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:31:24.454 INFO:journalctl@ceph.mon.a.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:31:24.454 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:31:24.454 INFO:journalctl@ceph.mon.a.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:31:24.454 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:31:24.454 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:31:24.454 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:31:24.454 INFO:journalctl@ceph.mon.a.smithi005.stdout: return future.result(timeout) 2024-03-22T05:31:24.454 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:31:24.454 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.__get_result() 2024-03-22T05:31:24.454 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:31:24.454 INFO:journalctl@ceph.mon.a.smithi005.stdout: raise self._exception 2024-03-22T05:31:24.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:31:24.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:31:24.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:31:24.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:31:24.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:31:24.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:31:24.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:31:24.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:31:24.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:31:24.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:31:24.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:31:24.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:31:24.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:31:24.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:31:24.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:31:24.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:31:24.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:31:24.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:31:24.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:31:24.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:31:24.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:31:24.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:31:24.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:31:24.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:31:24.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:31:24.457 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:24 smithi005 ceph-mon[29523]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:31:24.457 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:31:24.457 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:31:24.457 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:31:24.457 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:31:24.457 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:31:24.457 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:31:24.457 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:31:24.457 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:31:24.457 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:31:24.457 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:31:24.457 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:31:24.457 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:31:24.457 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:31:24.458 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:31:24.458 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:31:24.458 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:31:24.458 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:31:24.458 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:31:24.458 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:31:24.458 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:31:24.458 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:31:24.458 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:31:24.458 INFO:journalctl@ceph.mon.c.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:31:24.458 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:31:24.458 INFO:journalctl@ceph.mon.c.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:31:24.458 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:31:24.458 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:31:24.458 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:31:24.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: return future.result(timeout) 2024-03-22T05:31:24.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:31:24.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.__get_result() 2024-03-22T05:31:24.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:31:24.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: raise self._exception 2024-03-22T05:31:24.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:31:24.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:31:24.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:31:24.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:31:24.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:31:24.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:31:24.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:31:24.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:31:24.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:31:24.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:31:24.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:31:24.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:31:24.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:31:24.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:31:24.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:31:24.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:31:24.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:31:24.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:31:24.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:31:24.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:31:24.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:31:24.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:31:24.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:31:24.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:31:24.461 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:31:24.723 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:24 smithi186 ceph-mon[25720]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:31:24.723 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:31:24.723 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:31:24.724 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:31:24.724 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:31:24.724 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:31:24.724 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:31:24.724 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:31:24.724 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:31:24.724 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:31:24.724 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:31:24.724 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:31:24.724 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:31:24.724 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:31:24.724 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:31:24.724 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:31:24.724 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:31:24.725 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:31:24.725 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:31:24.725 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:31:24.725 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:31:24.725 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:31:24.725 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:31:24.725 INFO:journalctl@ceph.mon.b.smithi186.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:31:24.725 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:31:24.725 INFO:journalctl@ceph.mon.b.smithi186.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:31:24.725 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:31:24.725 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:31:24.725 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:31:24.726 INFO:journalctl@ceph.mon.b.smithi186.stdout: return future.result(timeout) 2024-03-22T05:31:24.726 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:31:24.726 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.__get_result() 2024-03-22T05:31:24.726 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:31:24.726 INFO:journalctl@ceph.mon.b.smithi186.stdout: raise self._exception 2024-03-22T05:31:24.726 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:31:24.726 INFO:journalctl@ceph.mon.b.smithi186.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:31:24.726 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:31:24.726 INFO:journalctl@ceph.mon.b.smithi186.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:31:24.726 INFO:journalctl@ceph.mon.b.smithi186.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:31:24.726 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:31:24.726 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:31:24.726 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:31:24.726 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:31:24.727 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:31:24.727 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:31:24.727 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:31:24.727 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:31:24.727 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:31:24.727 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:31:24.727 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:31:24.727 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:31:24.727 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:31:24.727 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:31:24.727 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:31:24.727 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:31:24.727 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:31:24.727 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:31:24.728 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:31:24.728 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:31:24.943 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:31:24 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:31:24.673+0000 7f84a310d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-22T05:31:24.943 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:31:24 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:31:24.942+0000 7f84a310d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-22T05:31:25.452 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:31:25 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:31:25.198+0000 7f84a310d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-22T05:31:25.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:25 smithi005 ceph-mon[24793]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 91 KiB/s rd, 0 B/s wr, 150 op/s 2024-03-22T05:31:25.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:25 smithi005 ceph-mon[29523]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 91 KiB/s rd, 0 B/s wr, 150 op/s 2024-03-22T05:31:25.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:25 smithi186 ceph-mon[25720]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 91 KiB/s rd, 0 B/s wr, 150 op/s 2024-03-22T05:31:26.344 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:31:26 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:31:26.086+0000 7f84a310d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-22T05:31:26.952 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:31:26 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:31:26.691+0000 7f84a310d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-22T05:31:26.952 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:31:26 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:31:26.805+0000 7f84a310d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-22T05:31:27.412 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:31:27 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:31:27.066+0000 7f84a310d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-22T05:31:27.412 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:31:27 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: [22/Mar/2024:05:31:27] ENGINE Bus STARTING 2024-03-22T05:31:27.412 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:31:27 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: CherryPy Checker: 2024-03-22T05:31:27.413 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:31:27 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: The Application mounted at '' has an empty config. 2024-03-22T05:31:27.413 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:31:27 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:31:27.413 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:31:27 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: [22/Mar/2024:05:31:27] ENGINE Serving on http://:::9283 2024-03-22T05:31:27.413 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:31:27 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: [22/Mar/2024:05:31:27] ENGINE Bus STARTED 2024-03-22T05:31:27.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:27 smithi005 ceph-mon[24793]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 72 op/s 2024-03-22T05:31:27.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:27 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:31:27.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:27 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:31:27.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:27 smithi005 ceph-mon[24793]: Standby manager daemon y restarted 2024-03-22T05:31:27.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:27 smithi005 ceph-mon[24793]: Standby manager daemon y started 2024-03-22T05:31:27.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:27 smithi005 ceph-mon[24793]: from='mgr.? 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-22T05:31:27.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:27 smithi005 ceph-mon[24793]: from='mgr.? 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-22T05:31:27.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:27 smithi005 ceph-mon[24793]: from='mgr.? 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-22T05:31:27.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:27 smithi005 ceph-mon[24793]: from='mgr.? 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-22T05:31:27.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:27 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:31:27.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:27 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:31:27.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:27 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:31:27.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:27 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:31:27.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:27 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:31:27.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:27 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:31:27.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:27 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:31:27.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:27 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:31:27.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:27 smithi005 ceph-mon[29523]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 72 op/s 2024-03-22T05:31:27.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:27 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:31:27.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:27 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:31:27.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:27 smithi005 ceph-mon[29523]: Standby manager daemon y restarted 2024-03-22T05:31:27.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:27 smithi005 ceph-mon[29523]: Standby manager daemon y started 2024-03-22T05:31:27.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:27 smithi005 ceph-mon[29523]: from='mgr.? 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-22T05:31:27.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:27 smithi005 ceph-mon[29523]: from='mgr.? 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-22T05:31:27.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:27 smithi005 ceph-mon[29523]: from='mgr.? 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-22T05:31:27.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:27 smithi005 ceph-mon[29523]: from='mgr.? 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-22T05:31:27.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:27 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:31:27.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:27 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:31:27.705 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:27 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:31:27.705 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:27 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:31:27.705 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:27 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:31:27.705 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:27 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:31:27.705 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:27 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:31:27.705 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:27 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:31:27.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:27 smithi186 ceph-mon[25720]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 72 op/s 2024-03-22T05:31:27.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:27 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:31:27.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:27 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:31:27.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:27 smithi186 ceph-mon[25720]: Standby manager daemon y restarted 2024-03-22T05:31:27.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:27 smithi186 ceph-mon[25720]: Standby manager daemon y started 2024-03-22T05:31:27.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:27 smithi186 ceph-mon[25720]: from='mgr.? 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-22T05:31:27.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:27 smithi186 ceph-mon[25720]: from='mgr.? 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-22T05:31:27.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:27 smithi186 ceph-mon[25720]: from='mgr.? 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-22T05:31:27.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:27 smithi186 ceph-mon[25720]: from='mgr.? 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-22T05:31:27.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:27 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:31:27.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:27 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:31:27.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:27 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:31:27.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:27 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:31:27.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:27 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:31:27.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:27 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:31:27.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:27 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:31:27.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:27 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:31:28.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:28 smithi005 ceph-mon[24793]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s rd, 0 B/s wr, 65 op/s 2024-03-22T05:31:28.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:28 smithi005 ceph-mon[24793]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 0 B/s wr, 79 op/s 2024-03-22T05:31:28.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:28 smithi005 ceph-mon[24793]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:31:28.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:28 smithi005 ceph-mon[24793]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:31:28.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:28 smithi005 ceph-mon[24793]: mgrmap e28: x(active, since 3m), standbys: y 2024-03-22T05:31:28.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:28 smithi005 ceph-mon[24793]: Health check update: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-22T05:31:28.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:28 smithi005 ceph-mon[29523]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s rd, 0 B/s wr, 65 op/s 2024-03-22T05:31:28.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:28 smithi005 ceph-mon[29523]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 0 B/s wr, 79 op/s 2024-03-22T05:31:28.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:28 smithi005 ceph-mon[29523]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:31:28.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:28 smithi005 ceph-mon[29523]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:31:28.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:28 smithi005 ceph-mon[29523]: mgrmap e28: x(active, since 3m), standbys: y 2024-03-22T05:31:28.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:28 smithi005 ceph-mon[29523]: Health check update: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-22T05:31:28.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:28 smithi186 ceph-mon[25720]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s rd, 0 B/s wr, 65 op/s 2024-03-22T05:31:28.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:28 smithi186 ceph-mon[25720]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 0 B/s wr, 79 op/s 2024-03-22T05:31:28.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:28 smithi186 ceph-mon[25720]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:31:28.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:28 smithi186 ceph-mon[25720]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:31:28.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:28 smithi186 ceph-mon[25720]: mgrmap e28: x(active, since 3m), standbys: y 2024-03-22T05:31:28.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:28 smithi186 ceph-mon[25720]: Health check update: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-22T05:31:28.794 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:31:28 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:31:28.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.5\", device_class=\"ssd\", hostname=\"smithi005\", instance=\"172.21.15.186:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.5\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"f59f36e2-e80a-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.5\", device_class=\"ssd\", hostname=\"smithi005\", instance=\"smithi186:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.5\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-22T05:31:30.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:30 smithi005 ceph-mon[29523]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 0 B/s wr, 79 op/s 2024-03-22T05:31:30.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:30 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:31:30.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:30 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:31:30.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:30 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:31:30.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:30 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-22T05:31:30.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:30 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-22T05:31:30.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:30 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:31:30.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:30 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/3380390106' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-22T05:31:30.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:30 smithi005 ceph-mon[24793]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 0 B/s wr, 79 op/s 2024-03-22T05:31:30.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:30 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:31:30.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:30 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:31:30.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:30 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:31:30.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:30 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-22T05:31:30.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:30 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-22T05:31:30.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:30 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:31:30.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:30 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/3380390106' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-22T05:31:30.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:30 smithi186 ceph-mon[25720]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 0 B/s wr, 79 op/s 2024-03-22T05:31:30.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:30 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:31:30.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:30 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:31:30.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:30 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:31:30.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:30 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-22T05:31:30.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:30 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-22T05:31:30.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:30 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:31:30.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:30 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/3380390106' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-22T05:31:31.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:31 smithi005 ceph-mon[24793]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-22T05:31:31.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:31 smithi005 ceph-mon[24793]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-22T05:31:31.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:31 smithi005 ceph-mon[29523]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-22T05:31:31.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:31 smithi005 ceph-mon[29523]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-22T05:31:31.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:31 smithi186 ceph-mon[25720]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-22T05:31:31.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:31 smithi186 ceph-mon[25720]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-22T05:31:31.794 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:31:31 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:31:31] "GET /metrics HTTP/1.1" 200 37958 "" "Prometheus/2.33.4" 2024-03-22T05:31:32.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:32 smithi005 ceph-mon[29523]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 338 B/s rd, 0 op/s 2024-03-22T05:31:32.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:32 smithi005 ceph-mon[24793]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 338 B/s rd, 0 op/s 2024-03-22T05:31:32.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:32 smithi186 ceph-mon[25720]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 338 B/s rd, 0 op/s 2024-03-22T05:31:33.794 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:31:33 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:31:33.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi005\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi005\", job=\"node\", machine=\"x86_64\", nodename=\"smithi005\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"f59f36e2-e80a-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi005\", job=\"node\", machine=\"x86_64\", nodename=\"smithi005\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-22T05:31:34.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:34 smithi005 ceph-mon[29523]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 338 B/s rd, 0 op/s 2024-03-22T05:31:34.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:34 smithi005 ceph-mon[24793]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 338 B/s rd, 0 op/s 2024-03-22T05:31:34.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:34 smithi186 ceph-mon[25720]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 338 B/s rd, 0 op/s 2024-03-22T05:31:35.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:35 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:31:35.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:35 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:31:35.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:35 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:31:35.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:35 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:31:35.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:35 smithi186 ceph-mon[25720]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 120 B/s rd, 0 op/s 2024-03-22T05:31:35.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:35 smithi186 ceph-mon[25720]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 137 B/s rd, 0 op/s 2024-03-22T05:31:35.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:35 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:31:35.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:35 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:31:35.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:35 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:31:35.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:35 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:31:35.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:35 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:31:35.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:35 smithi005 ceph-mon[24793]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 120 B/s rd, 0 op/s 2024-03-22T05:31:35.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:35 smithi005 ceph-mon[24793]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 137 B/s rd, 0 op/s 2024-03-22T05:31:35.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:35 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:31:35.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:35 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:31:35.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:35 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:31:35.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:35 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:31:35.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:35 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:31:35.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:35 smithi005 ceph-mon[29523]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 120 B/s rd, 0 op/s 2024-03-22T05:31:35.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:35 smithi005 ceph-mon[29523]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 137 B/s rd, 0 op/s 2024-03-22T05:31:35.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:35 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:31:36.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:36 smithi186 ceph-mon[25720]: Health check update: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-22T05:31:36.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:36 smithi005 ceph-mon[24793]: Health check update: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-22T05:31:36.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:36 smithi005 ceph-mon[29523]: Health check update: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-22T05:31:37.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:37 smithi005 ceph-mon[24793]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 650 B/s rd, 0 op/s 2024-03-22T05:31:37.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:37 smithi005 ceph-mon[29523]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 650 B/s rd, 0 op/s 2024-03-22T05:31:38.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:37 smithi186 ceph-mon[25720]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 650 B/s rd, 0 op/s 2024-03-22T05:31:39.044 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:31:38 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:31:38.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.5\", device_class=\"ssd\", hostname=\"smithi005\", instance=\"172.21.15.186:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.5\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"f59f36e2-e80a-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.5\", device_class=\"ssd\", hostname=\"smithi005\", instance=\"smithi186:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.5\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-22T05:31:39.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:39 smithi186 ceph-mon[25720]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 649 B/s rd, 0 op/s 2024-03-22T05:31:39.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:39 smithi005 ceph-mon[24793]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 649 B/s rd, 0 op/s 2024-03-22T05:31:39.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:39 smithi005 ceph-mon[29523]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 649 B/s rd, 0 op/s 2024-03-22T05:31:41.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:41 smithi186 ceph-mon[25720]: from='client.34937 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:31:41.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:41 smithi186 ceph-mon[25720]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 541 B/s rd, 0 op/s 2024-03-22T05:31:41.794 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:31:41 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:31:41] "GET /metrics HTTP/1.1" 200 37958 "" "Prometheus/2.33.4" 2024-03-22T05:31:41.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:41 smithi005 ceph-mon[24793]: from='client.34937 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:31:41.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:41 smithi005 ceph-mon[24793]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 541 B/s rd, 0 op/s 2024-03-22T05:31:41.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:41 smithi005 ceph-mon[29523]: from='client.34937 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:31:41.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:41 smithi005 ceph-mon[29523]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 541 B/s rd, 0 op/s 2024-03-22T05:31:43.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:43 smithi186 ceph-mon[25720]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:31:43.794 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:31:43 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:31:43.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi005\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi005\", job=\"node\", machine=\"x86_64\", nodename=\"smithi005\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"f59f36e2-e80a-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi005\", job=\"node\", machine=\"x86_64\", nodename=\"smithi005\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-22T05:31:43.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:43 smithi005 ceph-mon[24793]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:31:43.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:43 smithi005 ceph-mon[29523]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:31:45.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:45 smithi005 ceph-mon[24793]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:31:45.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:45 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:31:45.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:45 smithi005 ceph-mon[29523]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:31:45.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:45 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:31:46.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:45 smithi186 ceph-mon[25720]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:31:46.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:45 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:31:47.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:47 smithi005 ceph-mon[24793]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:31:47.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:47 smithi005 ceph-mon[29523]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:31:48.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:47 smithi186 ceph-mon[25720]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:31:49.043 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:31:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:31:48.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.5\", device_class=\"ssd\", hostname=\"smithi005\", instance=\"172.21.15.186:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.5\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"f59f36e2-e80a-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.5\", device_class=\"ssd\", hostname=\"smithi005\", instance=\"smithi186:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.5\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-22T05:31:49.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:49 smithi186 ceph-mon[25720]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:31:49.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:49 smithi005 ceph-mon[24793]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:31:49.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:49 smithi005 ceph-mon[29523]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:31:51.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:51 smithi186 ceph-mon[25720]: from='client.34937 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:31:51.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:51 smithi186 ceph-mon[25720]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:31:51.794 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:31:51 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:31:51] "GET /metrics HTTP/1.1" 200 37961 "" "Prometheus/2.33.4" 2024-03-22T05:31:51.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:51 smithi005 ceph-mon[24793]: from='client.34937 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:31:51.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:51 smithi005 ceph-mon[24793]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:31:51.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:51 smithi005 ceph-mon[29523]: from='client.34937 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:31:51.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:51 smithi005 ceph-mon[29523]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:31:53.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:53 smithi186 ceph-mon[25720]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:31:53.794 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:31:53 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:31:53.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi005\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi005\", job=\"node\", machine=\"x86_64\", nodename=\"smithi005\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"f59f36e2-e80a-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi005\", job=\"node\", machine=\"x86_64\", nodename=\"smithi005\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-22T05:31:53.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:53 smithi005 ceph-mon[24793]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:31:53.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:53 smithi005 ceph-mon[29523]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:31:55.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:55 smithi005 ceph-mon[29523]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:31:55.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:55 smithi005 ceph-mon[24793]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:31:56.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:55 smithi186 ceph-mon[25720]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:31:57.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:57 smithi005 ceph-mon[24793]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:31:57.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:57 smithi005 ceph-mon[29523]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:31:58.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:57 smithi186 ceph-mon[25720]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:31:59.044 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:31:58 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:31:58.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.5\", device_class=\"ssd\", hostname=\"smithi005\", instance=\"172.21.15.186:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.5\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"f59f36e2-e80a-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.5\", device_class=\"ssd\", hostname=\"smithi005\", instance=\"smithi186:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.5\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-22T05:31:59.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:31:59 smithi186 ceph-mon[25720]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:31:59.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:31:59 smithi005 ceph-mon[29523]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:31:59.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:31:59 smithi005 ceph-mon[24793]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:32:00.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:32:00 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:32:00.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:32:00 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:32:00.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:32:00 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:32:01.544 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:32:01 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:32:01] "GET /metrics HTTP/1.1" 200 37961 "" "Prometheus/2.33.4" 2024-03-22T05:32:01.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:32:01 smithi186 ceph-mon[25720]: from='client.34937 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:32:01.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:32:01 smithi186 ceph-mon[25720]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:32:01.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:32:01 smithi005 ceph-mon[24793]: from='client.34937 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:32:01.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:32:01 smithi005 ceph-mon[24793]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:32:01.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:32:01 smithi005 ceph-mon[29523]: from='client.34937 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:32:01.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:32:01 smithi005 ceph-mon[29523]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:32:03.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:32:03 smithi186 ceph-mon[25720]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:32:03.794 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:32:03 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:32:03.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi005\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi005\", job=\"node\", machine=\"x86_64\", nodename=\"smithi005\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"f59f36e2-e80a-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi005\", job=\"node\", machine=\"x86_64\", nodename=\"smithi005\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-22T05:32:03.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:32:03 smithi005 ceph-mon[24793]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:32:03.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:32:03 smithi005 ceph-mon[29523]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:32:05.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:32:05 smithi005 ceph-mon[24793]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:32:05.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:32:05 smithi005 ceph-mon[29523]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:32:06.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:32:05 smithi186 ceph-mon[25720]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:32:07.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:32:07 smithi005 ceph-mon[24793]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:32:07.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:32:07 smithi005 ceph-mon[29523]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:32:08.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:32:07 smithi186 ceph-mon[25720]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:32:09.043 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:32:08 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:32:08.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.5\", device_class=\"ssd\", hostname=\"smithi005\", instance=\"172.21.15.186:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.5\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"f59f36e2-e80a-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.5\", device_class=\"ssd\", hostname=\"smithi005\", instance=\"smithi186:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.5\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-22T05:32:09.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:32:09 smithi186 ceph-mon[25720]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:32:09.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:32:09 smithi005 ceph-mon[24793]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:32:09.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:32:09 smithi005 ceph-mon[29523]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:32:11.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:32:11 smithi186 ceph-mon[25720]: from='client.34937 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:32:11.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:32:11 smithi186 ceph-mon[25720]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:32:11.794 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:32:11 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:32:11] "GET /metrics HTTP/1.1" 200 37961 "" "Prometheus/2.33.4" 2024-03-22T05:32:11.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:32:11 smithi005 ceph-mon[24793]: from='client.34937 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:32:11.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:32:11 smithi005 ceph-mon[24793]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:32:11.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:32:11 smithi005 ceph-mon[29523]: from='client.34937 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:32:11.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:32:11 smithi005 ceph-mon[29523]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:32:13.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:32:13 smithi186 ceph-mon[25720]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:32:13.794 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:32:13 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:32:13.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi005\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi005\", job=\"node\", machine=\"x86_64\", nodename=\"smithi005\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"f59f36e2-e80a-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi005\", job=\"node\", machine=\"x86_64\", nodename=\"smithi005\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-22T05:32:13.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:32:13 smithi005 ceph-mon[29523]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:32:13.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:32:13 smithi005 ceph-mon[24793]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:32:15.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:32:15 smithi005 ceph-mon[29523]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:32:15.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:32:15 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:32:15.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:32:15 smithi005 ceph-mon[24793]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:32:15.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:32:15 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:32:16.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:32:15 smithi186 ceph-mon[25720]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:32:16.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:32:15 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:32:17.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:32:17 smithi005 ceph-mon[29523]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:32:17.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:32:17 smithi005 ceph-mon[24793]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:32:18.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:32:17 smithi186 ceph-mon[25720]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:32:19.044 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:32:18 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:32:18.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.5\", device_class=\"ssd\", hostname=\"smithi005\", instance=\"172.21.15.186:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.5\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"f59f36e2-e80a-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.5\", device_class=\"ssd\", hostname=\"smithi005\", instance=\"smithi186:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.5\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-22T05:32:19.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:32:19 smithi186 ceph-mon[25720]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:32:19.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:32:19 smithi005 ceph-mon[24793]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:32:19.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:32:19 smithi005 ceph-mon[29523]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:32:21.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:32:21 smithi186 ceph-mon[25720]: from='client.34937 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:32:21.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:32:21 smithi186 ceph-mon[25720]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:32:21.794 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:32:21 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:32:21] "GET /metrics HTTP/1.1" 200 37961 "" "Prometheus/2.33.4" 2024-03-22T05:32:21.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:32:21 smithi005 ceph-mon[24793]: from='client.34937 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:32:21.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:32:21 smithi005 ceph-mon[24793]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:32:21.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:32:21 smithi005 ceph-mon[29523]: from='client.34937 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:32:21.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:32:21 smithi005 ceph-mon[29523]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:32:23.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:32:23 smithi186 ceph-mon[25720]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:32:23.794 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:32:23 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:32:23.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi005\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi005\", job=\"node\", machine=\"x86_64\", nodename=\"smithi005\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"f59f36e2-e80a-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi005\", job=\"node\", machine=\"x86_64\", nodename=\"smithi005\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-22T05:32:23.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:32:23 smithi005 ceph-mon[24793]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:32:23.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:32:23 smithi005 ceph-mon[29523]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:32:25.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:32:25 smithi005 ceph-mon[29523]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:32:25.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:32:25 smithi005 ceph-mon[24793]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:32:26.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:32:25 smithi186 ceph-mon[25720]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:32:27.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:32:27 smithi005 ceph-mon[29523]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:32:27.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:32:27 smithi005 ceph-mon[24793]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:32:28.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:32:27 smithi186 ceph-mon[25720]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:32:29.044 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:32:28 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:32:28.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.5\", device_class=\"ssd\", hostname=\"smithi005\", instance=\"172.21.15.186:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.5\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"f59f36e2-e80a-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.5\", device_class=\"ssd\", hostname=\"smithi005\", instance=\"smithi186:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.5\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-22T05:32:29.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:32:29 smithi186 ceph-mon[25720]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:32:29.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:32:29 smithi005 ceph-mon[29523]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:32:29.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:32:29 smithi005 ceph-mon[24793]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:32:30.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:32:30 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:32:30.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:32:30 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:32:30.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:32:30 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:32:31.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:32:31 smithi186 ceph-mon[25720]: from='client.34937 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:32:31.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:32:31 smithi186 ceph-mon[25720]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:32:31.544 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:32:31 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:32:31] "GET /metrics HTTP/1.1" 200 37960 "" "Prometheus/2.33.4" 2024-03-22T05:32:31.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:32:31 smithi005 ceph-mon[29523]: from='client.34937 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:32:31.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:32:31 smithi005 ceph-mon[29523]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:32:31.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:32:31 smithi005 ceph-mon[24793]: from='client.34937 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:32:31.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:32:31 smithi005 ceph-mon[24793]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:32:33.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:32:33 smithi186 ceph-mon[25720]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:32:33.794 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:32:33 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:32:33.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi005\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi005\", job=\"node\", machine=\"x86_64\", nodename=\"smithi005\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"f59f36e2-e80a-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi005\", job=\"node\", machine=\"x86_64\", nodename=\"smithi005\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-22T05:32:33.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:32:33 smithi005 ceph-mon[29523]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:32:33.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:32:33 smithi005 ceph-mon[24793]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:32:35.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:32:35 smithi005 ceph-mon[24793]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:32:35.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:32:35 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:32:35.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:32:35 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:32:35.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:32:35 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:32:35.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:32:35 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:32:35.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:32:35 smithi005 ceph-mon[29523]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:32:35.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:32:35 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:32:35.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:32:35 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:32:35.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:32:35 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:32:35.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:32:35 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:32:36.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:32:35 smithi186 ceph-mon[25720]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:32:36.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:32:35 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:32:36.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:32:35 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:32:36.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:32:35 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:32:36.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:32:35 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:32:36.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:32:36 smithi005 ceph-mon[24793]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 948 B/s rd, 0 op/s 2024-03-22T05:32:36.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:32:36 smithi005 ceph-mon[24793]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 581 B/s rd, 0 op/s 2024-03-22T05:32:36.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:32:36 smithi005 ceph-mon[29523]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 948 B/s rd, 0 op/s 2024-03-22T05:32:36.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:32:36 smithi005 ceph-mon[29523]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 581 B/s rd, 0 op/s 2024-03-22T05:32:37.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:32:36 smithi186 ceph-mon[25720]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 948 B/s rd, 0 op/s 2024-03-22T05:32:37.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:32:36 smithi186 ceph-mon[25720]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 581 B/s rd, 0 op/s 2024-03-22T05:32:38.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:32:38 smithi005 ceph-mon[29523]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:32:38.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:32:38 smithi005 ceph-mon[24793]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:32:39.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:32:38 smithi186 ceph-mon[25720]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:32:39.044 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:32:38 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:32:38.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.5\", device_class=\"ssd\", hostname=\"smithi005\", instance=\"172.21.15.186:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.5\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"f59f36e2-e80a-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.5\", device_class=\"ssd\", hostname=\"smithi005\", instance=\"smithi186:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.5\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-22T05:32:40.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:32:40 smithi005 ceph-mon[29523]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:32:40.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:32:40 smithi005 ceph-mon[24793]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:32:41.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:32:40 smithi186 ceph-mon[25720]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:32:41.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:32:41 smithi186 ceph-mon[25720]: from='client.34937 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:32:41.794 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:32:41 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:32:41] "GET /metrics HTTP/1.1" 200 37960 "" "Prometheus/2.33.4" 2024-03-22T05:32:41.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:32:41 smithi005 ceph-mon[29523]: from='client.34937 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:32:41.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:32:41 smithi005 ceph-mon[24793]: from='client.34937 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:32:42.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:32:42 smithi005 ceph-mon[24793]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:32:42.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:32:42 smithi005 ceph-mon[29523]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:32:43.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:32:42 smithi186 ceph-mon[25720]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:32:43.793 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:32:43 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:32:43.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi005\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi005\", job=\"node\", machine=\"x86_64\", nodename=\"smithi005\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"f59f36e2-e80a-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi005\", job=\"node\", machine=\"x86_64\", nodename=\"smithi005\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-22T05:32:44.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:32:44 smithi186 ceph-mon[25720]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:32:44.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:32:44 smithi005 ceph-mon[24793]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:32:44.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:32:44 smithi005 ceph-mon[29523]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:32:45.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:32:45 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:32:45.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:32:45 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:32:45.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:32:45 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:32:46.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:32:46 smithi186 ceph-mon[25720]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:32:46.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:32:46 smithi005 ceph-mon[24793]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:32:46.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:32:46 smithi005 ceph-mon[29523]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:32:48.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:32:48 smithi005 ceph-mon[24793]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:32:48.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:32:48 smithi005 ceph-mon[29523]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:32:48.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:32:48 smithi186 ceph-mon[25720]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:32:48.794 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:32:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:32:48.661Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.5\", device_class=\"ssd\", hostname=\"smithi005\", instance=\"172.21.15.186:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.5\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"f59f36e2-e80a-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.5\", device_class=\"ssd\", hostname=\"smithi005\", instance=\"smithi186:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.5\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-22T05:32:50.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:32:50 smithi005 ceph-mon[24793]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:32:50.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:32:50 smithi005 ceph-mon[29523]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:32:50.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:32:50 smithi186 ceph-mon[25720]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:32:51.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:32:51 smithi005 ceph-mon[24793]: from='client.34937 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:32:51.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:32:51 smithi005 ceph-mon[29523]: from='client.34937 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:32:51.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:32:51 smithi186 ceph-mon[25720]: from='client.34937 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:32:51.794 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:32:51 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:32:51] "GET /metrics HTTP/1.1" 200 37961 "" "Prometheus/2.33.4" 2024-03-22T05:32:52.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:32:52 smithi005 ceph-mon[24793]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:32:52.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:32:52 smithi005 ceph-mon[29523]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:32:52.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:32:52 smithi186 ceph-mon[25720]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:32:53.794 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:32:53 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:32:53.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi005\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi005\", job=\"node\", machine=\"x86_64\", nodename=\"smithi005\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"f59f36e2-e80a-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi005\", job=\"node\", machine=\"x86_64\", nodename=\"smithi005\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-22T05:32:54.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:32:54 smithi186 ceph-mon[25720]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:32:54.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:32:54 smithi005 ceph-mon[24793]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:32:54.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:32:54 smithi005 ceph-mon[29523]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:32:56.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:32:56 smithi005 ceph-mon[24793]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:32:56.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:32:56 smithi005 ceph-mon[29523]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:32:56.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:32:56 smithi186 ceph-mon[25720]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:32:58.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:32:58 smithi005 ceph-mon[24793]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:32:58.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:32:58 smithi005 ceph-mon[29523]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:32:58.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:32:58 smithi186 ceph-mon[25720]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:32:58.794 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:32:58 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:32:58.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.5\", device_class=\"ssd\", hostname=\"smithi005\", instance=\"172.21.15.186:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.5\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"f59f36e2-e80a-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.5\", device_class=\"ssd\", hostname=\"smithi005\", instance=\"smithi186:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.5\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-22T05:33:00.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:33:00 smithi005 ceph-mon[24793]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:33:00.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:33:00 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:33:00.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:33:00 smithi005 ceph-mon[29523]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:33:00.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:33:00 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:33:00.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:33:00 smithi186 ceph-mon[25720]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:33:00.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:33:00 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:33:01.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:33:01 smithi005 ceph-mon[24793]: from='client.34937 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:33:01.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:33:01 smithi005 ceph-mon[29523]: from='client.34937 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:33:01.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:33:01 smithi186 ceph-mon[25720]: from='client.34937 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:33:01.794 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:33:01 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:33:01] "GET /metrics HTTP/1.1" 200 37960 "" "Prometheus/2.33.4" 2024-03-22T05:33:02.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:33:02 smithi005 ceph-mon[29523]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:33:02.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:33:02 smithi005 ceph-mon[24793]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:33:02.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:33:02 smithi186 ceph-mon[25720]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:33:03.793 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:33:03 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:33:03.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi005\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi005\", job=\"node\", machine=\"x86_64\", nodename=\"smithi005\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"f59f36e2-e80a-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi005\", job=\"node\", machine=\"x86_64\", nodename=\"smithi005\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-22T05:33:04.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:33:04 smithi186 ceph-mon[25720]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:33:04.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:33:04 smithi005 ceph-mon[24793]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:33:04.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:33:04 smithi005 ceph-mon[29523]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:33:06.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:33:06 smithi005 ceph-mon[29523]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:33:06.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:33:06 smithi005 ceph-mon[24793]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:33:06.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:33:06 smithi186 ceph-mon[25720]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:33:08.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:33:08 smithi005 ceph-mon[29523]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:33:08.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:33:08 smithi005 ceph-mon[24793]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:33:08.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:33:08 smithi186 ceph-mon[25720]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:33:08.794 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:33:08 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:33:08.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.5\", device_class=\"ssd\", hostname=\"smithi005\", instance=\"172.21.15.186:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.5\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"f59f36e2-e80a-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.5\", device_class=\"ssd\", hostname=\"smithi005\", instance=\"smithi186:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.5\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-22T05:33:10.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:33:10 smithi005 ceph-mon[24793]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:33:10.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:33:10 smithi005 ceph-mon[29523]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:33:10.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:33:10 smithi186 ceph-mon[25720]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:33:11.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:33:11 smithi005 ceph-mon[24793]: from='client.34937 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:33:11.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:33:11 smithi005 ceph-mon[29523]: from='client.34937 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:33:11.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:33:11 smithi186 ceph-mon[25720]: from='client.34937 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:33:11.794 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:33:11 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:33:11] "GET /metrics HTTP/1.1" 200 37960 "" "Prometheus/2.33.4" 2024-03-22T05:33:12.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:33:12 smithi005 ceph-mon[24793]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:33:12.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:33:12 smithi005 ceph-mon[29523]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:33:12.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:33:12 smithi186 ceph-mon[25720]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:33:13.794 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:33:13 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:33:13.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi005\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi005\", job=\"node\", machine=\"x86_64\", nodename=\"smithi005\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"f59f36e2-e80a-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi005\", job=\"node\", machine=\"x86_64\", nodename=\"smithi005\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-22T05:33:14.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:33:14 smithi186 ceph-mon[25720]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:33:14.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:33:14 smithi005 ceph-mon[29523]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:33:14.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:33:14 smithi005 ceph-mon[24793]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:33:15.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:33:15 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:33:15.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:33:15 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:33:15.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:33:15 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:33:16.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:33:16 smithi186 ceph-mon[25720]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:33:16.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:33:16 smithi005 ceph-mon[29523]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:33:16.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:33:16 smithi005 ceph-mon[24793]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:33:18.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:33:18 smithi005 ceph-mon[29523]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:33:18.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:33:18 smithi005 ceph-mon[24793]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:33:18.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:33:18 smithi186 ceph-mon[25720]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:33:18.794 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:33:18 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:33:18.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.5\", device_class=\"ssd\", hostname=\"smithi005\", instance=\"172.21.15.186:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.5\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"f59f36e2-e80a-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.5\", device_class=\"ssd\", hostname=\"smithi005\", instance=\"smithi186:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.5\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-22T05:33:20.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:33:20 smithi005 ceph-mon[24793]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:33:20.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:33:20 smithi005 ceph-mon[29523]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:33:20.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:33:20 smithi186 ceph-mon[25720]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:33:21.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:33:21 smithi005 ceph-mon[24793]: from='client.34937 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:33:21.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:33:21 smithi005 ceph-mon[29523]: from='client.34937 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:33:21.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:33:21 smithi186 ceph-mon[25720]: from='client.34937 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:33:21.794 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:33:21 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:33:21] "GET /metrics HTTP/1.1" 200 37960 "" "Prometheus/2.33.4" 2024-03-22T05:33:22.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:33:22 smithi005 ceph-mon[24793]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:33:22.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:33:22 smithi005 ceph-mon[29523]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:33:22.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:33:22 smithi186 ceph-mon[25720]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:33:23.793 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:33:23 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:33:23.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi005\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi005\", job=\"node\", machine=\"x86_64\", nodename=\"smithi005\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"f59f36e2-e80a-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi005\", job=\"node\", machine=\"x86_64\", nodename=\"smithi005\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-22T05:33:24.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:33:24 smithi186 ceph-mon[25720]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:33:24.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:33:24 smithi005 ceph-mon[24793]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:33:24.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:33:24 smithi005 ceph-mon[29523]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:33:26.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:33:26 smithi005 ceph-mon[24793]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:33:26.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:33:26 smithi005 ceph-mon[29523]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:33:26.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:33:26 smithi186 ceph-mon[25720]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:33:28.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:33:28 smithi005 ceph-mon[24793]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:33:28.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:33:28 smithi005 ceph-mon[29523]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:33:28.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:33:28 smithi186 ceph-mon[25720]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:33:28.794 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:33:28 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:33:28.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.5\", device_class=\"ssd\", hostname=\"smithi005\", instance=\"172.21.15.186:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.5\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"f59f36e2-e80a-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.5\", device_class=\"ssd\", hostname=\"smithi005\", instance=\"smithi186:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.5\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-22T05:33:30.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:33:30 smithi005 ceph-mon[24793]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:33:30.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:33:30 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:33:30.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:33:30 smithi005 ceph-mon[29523]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:33:30.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:33:30 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:33:30.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:33:30 smithi186 ceph-mon[25720]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:33:30.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:33:30 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:33:31.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:33:31 smithi005 ceph-mon[24793]: from='client.34937 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:33:31.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:33:31 smithi005 ceph-mon[29523]: from='client.34937 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:33:31.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:33:31 smithi186 ceph-mon[25720]: from='client.34937 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:33:31.794 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:33:31 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:33:31] "GET /metrics HTTP/1.1" 200 37960 "" "Prometheus/2.33.4" 2024-03-22T05:33:32.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:33:32 smithi005 ceph-mon[24793]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:33:32.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:33:32 smithi005 ceph-mon[29523]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:33:32.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:33:32 smithi186 ceph-mon[25720]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:33:34.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:33:34 smithi186 ceph-mon[25720]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:33:34.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:33:34 smithi005 ceph-mon[24793]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:33:34.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:33:34 smithi005 ceph-mon[29523]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:33:36.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:33:36 smithi005 ceph-mon[29523]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:33:36.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:33:36 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:33:36.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:33:36 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:33:36.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:33:36 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:33:36.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:33:36 smithi005 ceph-mon[29523]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:33:36.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:33:36 smithi005 ceph-mon[24793]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:33:36.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:33:36 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:33:36.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:33:36 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:33:36.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:33:36 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:33:36.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:33:36 smithi005 ceph-mon[24793]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:33:36.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:33:36 smithi186 ceph-mon[25720]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:33:36.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:33:36 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:33:36.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:33:36 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:33:36.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:33:36 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:33:36.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:33:36 smithi186 ceph-mon[25720]: from='mgr.34760 ' entity='mgr.x' 2024-03-22T05:33:37.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:33:37 smithi005 ceph-mon[29523]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-22T05:33:37.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:33:37 smithi005 ceph-mon[29523]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:33:37.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:33:37 smithi005 ceph-mon[24793]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-22T05:33:37.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:33:37 smithi005 ceph-mon[24793]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:33:37.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:33:37 smithi186 ceph-mon[25720]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-22T05:33:37.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:33:37 smithi186 ceph-mon[25720]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:33:39.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:33:39 smithi005 ceph-mon[24793]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:33:39.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:33:39 smithi005 ceph-mon[29523]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:33:39.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:33:39 smithi186 ceph-mon[25720]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:33:41.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:33:41 smithi005 ceph-mon[24793]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 589 B/s rd, 0 op/s 2024-03-22T05:33:41.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:33:41 smithi005 ceph-mon[24793]: from='client.34937 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:33:41.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:33:41 smithi005 ceph-mon[29523]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 589 B/s rd, 0 op/s 2024-03-22T05:33:41.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:33:41 smithi005 ceph-mon[29523]: from='client.34937 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:33:41.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:33:41 smithi186 ceph-mon[25720]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 589 B/s rd, 0 op/s 2024-03-22T05:33:41.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:33:41 smithi186 ceph-mon[25720]: from='client.34937 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:33:41.794 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:33:41 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:33:41] "GET /metrics HTTP/1.1" 200 37960 "" "Prometheus/2.33.4" 2024-03-22T05:33:43.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:33:43 smithi186 ceph-mon[25720]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:33:43.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:33:43 smithi005 ceph-mon[29523]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:33:43.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:33:43 smithi005 ceph-mon[24793]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:33:45.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:33:45 smithi186 ceph-mon[25720]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:33:45.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:33:45 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:33:45.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:33:45 smithi005 ceph-mon[29523]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:33:45.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:33:45 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:33:45.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:33:45 smithi005 ceph-mon[24793]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:33:45.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:33:45 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:33:47.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:33:47 smithi186 ceph-mon[25720]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:33:47.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:33:47 smithi005 ceph-mon[24793]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:33:47.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:33:47 smithi005 ceph-mon[29523]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:33:49.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:33:49 smithi186 ceph-mon[25720]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:33:49.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:33:49 smithi005 ceph-mon[24793]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:33:49.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:33:49 smithi005 ceph-mon[29523]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:33:51.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:33:51 smithi186 ceph-mon[25720]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:33:51.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:33:51 smithi186 ceph-mon[25720]: from='client.34937 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:33:51.794 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:33:51 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:33:51] "GET /metrics HTTP/1.1" 200 37956 "" "Prometheus/2.33.4" 2024-03-22T05:33:51.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:33:51 smithi005 ceph-mon[29523]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:33:51.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:33:51 smithi005 ceph-mon[29523]: from='client.34937 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:33:51.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:33:51 smithi005 ceph-mon[24793]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:33:51.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:33:51 smithi005 ceph-mon[24793]: from='client.34937 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:33:53.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:33:53 smithi186 ceph-mon[25720]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:33:53.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:33:53 smithi005 ceph-mon[24793]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:33:53.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:33:53 smithi005 ceph-mon[29523]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:33:55.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:33:55 smithi186 ceph-mon[25720]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:33:55.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:33:55 smithi005 ceph-mon[24793]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:33:55.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:33:55 smithi005 ceph-mon[29523]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:33:57.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:33:57 smithi186 ceph-mon[25720]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:33:57.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:33:57 smithi005 ceph-mon[29523]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:33:57.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:33:57 smithi005 ceph-mon[24793]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:33:59.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:33:59 smithi186 ceph-mon[25720]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:33:59.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:33:59 smithi005 ceph-mon[29523]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:33:59.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:33:59 smithi005 ceph-mon[24793]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:34:00.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:00 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:34:00.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:00 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:34:00.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:00 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:34:01.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:01 smithi186 ceph-mon[25720]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:34:01.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:01 smithi186 ceph-mon[25720]: from='client.34937 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:34:01.794 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:34:01 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:34:01] "GET /metrics HTTP/1.1" 200 37959 "" "Prometheus/2.33.4" 2024-03-22T05:34:01.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:01 smithi005 ceph-mon[24793]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:34:01.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:01 smithi005 ceph-mon[24793]: from='client.34937 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:34:01.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:01 smithi005 ceph-mon[29523]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:34:01.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:01 smithi005 ceph-mon[29523]: from='client.34937 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:34:03.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:03 smithi186 ceph-mon[25720]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:34:03.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:03 smithi005 ceph-mon[24793]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:34:03.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:03 smithi005 ceph-mon[29523]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:34:05.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:05 smithi186 ceph-mon[25720]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:34:05.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:05 smithi005 ceph-mon[24793]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:34:05.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:05 smithi005 ceph-mon[29523]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:34:07.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:07 smithi186 ceph-mon[25720]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:34:07.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:07 smithi005 ceph-mon[24793]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:34:07.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:07 smithi005 ceph-mon[29523]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:34:09.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:09 smithi186 ceph-mon[25720]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:34:09.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:09 smithi005 ceph-mon[24793]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:34:09.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:09 smithi005 ceph-mon[29523]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:34:11.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:11 smithi186 ceph-mon[25720]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:34:11.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:11 smithi186 ceph-mon[25720]: from='client.34937 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:34:11.794 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:34:11 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:34:11] "GET /metrics HTTP/1.1" 200 37959 "" "Prometheus/2.33.4" 2024-03-22T05:34:11.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:11 smithi005 ceph-mon[24793]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:34:11.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:11 smithi005 ceph-mon[24793]: from='client.34937 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:34:11.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:11 smithi005 ceph-mon[29523]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:34:11.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:11 smithi005 ceph-mon[29523]: from='client.34937 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:34:12.948 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph orch ps' 2024-03-22T05:34:13.517 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:13 smithi005 ceph-mon[24793]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:34:13.517 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:13 smithi005 ceph-mon[29523]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:34:13.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:13 smithi186 ceph-mon[25720]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:34:15.086 INFO:teuthology.orchestra.run.smithi005.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T05:34:15.086 INFO:teuthology.orchestra.run.smithi005.stdout:alertmanager.a smithi005 *:9093,9094 running (5m) 2m ago 13m 25.7M - 0.23.0 ba2b418f427c fd7dc5dbe3e5 2024-03-22T05:34:15.086 INFO:teuthology.orchestra.run.smithi005.stdout:grafana.a smithi186 *:3000 error 2m ago 13m - - 2024-03-22T05:34:15.086 INFO:teuthology.orchestra.run.smithi005.stdout:iscsi.foo.smithi005.mcdwbp smithi005 running (2m) 2m ago 12m 45.9M - 3.5 e1d6a67b021e 9d19a0c36be9 2024-03-22T05:34:15.086 INFO:teuthology.orchestra.run.smithi005.stdout:mgr.x smithi186 *:8443,9283 running (8m) 2m ago 18m 511M - 19.0.0-2299-gea01217f aa461e7c5e7f 66924e151e7b 2024-03-22T05:34:15.086 INFO:teuthology.orchestra.run.smithi005.stdout:mgr.y smithi005 *:8443,9283,8765 running (2m) 2m ago 19m 448M - 19.0.0-2299-gea01217f aa461e7c5e7f 62a813eebc0d 2024-03-22T05:34:15.086 INFO:teuthology.orchestra.run.smithi005.stdout:mon.a smithi005 running (19m) 2m ago 19m 59.7M 2048M 17.2.0 e1d6a67b021e 38e0bba4bb03 2024-03-22T05:34:15.086 INFO:teuthology.orchestra.run.smithi005.stdout:mon.b smithi186 running (18m) 2m ago 18m 46.9M 2048M 17.2.0 e1d6a67b021e 323f53f932f6 2024-03-22T05:34:15.087 INFO:teuthology.orchestra.run.smithi005.stdout:mon.c smithi005 running (18m) 2m ago 18m 46.2M 2048M 17.2.0 e1d6a67b021e 00ec24d38612 2024-03-22T05:34:15.087 INFO:teuthology.orchestra.run.smithi005.stdout:node-exporter.a smithi005 *:9100 running (6m) 2m ago 14m 21.0M - 1.3.1 1dbe0e931976 a9085d6042eb 2024-03-22T05:34:15.087 INFO:teuthology.orchestra.run.smithi005.stdout:node-exporter.b smithi186 *:9100 running (5m) 2m ago 14m 18.8M - 1.3.1 1dbe0e931976 74aa1181ff71 2024-03-22T05:34:15.087 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0 smithi005 running (17m) 2m ago 17m 52.1M 2993M 17.2.0 e1d6a67b021e ef5c7053dd37 2024-03-22T05:34:15.087 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1 smithi005 running (17m) 2m ago 17m 54.8M 2993M 17.2.0 e1d6a67b021e a1e7ee1017b6 2024-03-22T05:34:15.087 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2 smithi005 running (17m) 2m ago 17m 47.7M 2993M 17.2.0 e1d6a67b021e 462ea63ad0e5 2024-03-22T05:34:15.087 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3 smithi005 running (16m) 2m ago 16m 54.9M 2993M 17.2.0 e1d6a67b021e b25432a7239b 2024-03-22T05:34:15.087 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4 smithi186 running (16m) 2m ago 16m 53.1M 3248M 17.2.0 e1d6a67b021e cd963671f340 2024-03-22T05:34:15.087 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5 smithi186 running (15m) 2m ago 15m 50.1M 3248M 17.2.0 e1d6a67b021e f98c87f1a2cd 2024-03-22T05:34:15.087 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6 smithi186 running (15m) 2m ago 15m 49.1M 3248M 17.2.0 e1d6a67b021e 704ac49c68c4 2024-03-22T05:34:15.087 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7 smithi186 running (15m) 2m ago 15m 51.8M 3248M 17.2.0 e1d6a67b021e f415b90edf63 2024-03-22T05:34:15.087 INFO:teuthology.orchestra.run.smithi005.stdout:prometheus.a smithi186 *:9095 running (5m) 2m ago 14m 68.6M - 2.33.4 514e6a882f6e c3a69c1f7e79 2024-03-22T05:34:15.087 INFO:teuthology.orchestra.run.smithi005.stdout:rgw.foo.smithi005.exdhqu smithi005 *:8000 running (13m) 2m ago 13m 75.6M - 17.2.0 e1d6a67b021e 6bb27fe11553 2024-03-22T05:34:15.087 INFO:teuthology.orchestra.run.smithi005.stdout:rgw.foo.smithi186.mcgodk smithi186 *:8000 running (12m) 2m ago 12m 73.3M - 17.2.0 e1d6a67b021e 19b5c20a34f7 2024-03-22T05:34:15.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:15 smithi005 ceph-mon[24793]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:34:15.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:15 smithi005 ceph-mon[24793]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:34:15.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:15 smithi005 ceph-mon[29523]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:34:15.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:15 smithi005 ceph-mon[29523]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:34:15.730 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph versions' 2024-03-22T05:34:15.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:15 smithi186 ceph-mon[25720]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:34:15.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:15 smithi186 ceph-mon[25720]: from='mgr.34760 172.21.15.186:0/246746181' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:34:16.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:16 smithi005 ceph-mon[29523]: from='client.25141 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:34:16.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:16 smithi005 ceph-mon[24793]: from='client.25141 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:34:16.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:16 smithi186 ceph-mon[25720]: from='client.25141 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:34:17.635 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:17 smithi005 ceph-mon[24793]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:34:17.635 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:17 smithi005 ceph-mon[29523]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:34:17.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:17 smithi186 ceph-mon[25720]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:34:17.968 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-03-22T05:34:17.968 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": { 2024-03-22T05:34:17.968 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-03-22T05:34:17.968 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:34:17.968 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr": { 2024-03-22T05:34:17.969 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev)": 2 2024-03-22T05:34:17.969 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:34:17.969 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": { 2024-03-22T05:34:17.969 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-22T05:34:17.969 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:34:17.969 INFO:teuthology.orchestra.run.smithi005.stdout: "mds": {}, 2024-03-22T05:34:17.969 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw": { 2024-03-22T05:34:17.969 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-22T05:34:17.969 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:34:17.969 INFO:teuthology.orchestra.run.smithi005.stdout: "overall": { 2024-03-22T05:34:17.969 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-03-22T05:34:17.969 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev)": 2 2024-03-22T05:34:17.969 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-03-22T05:34:17.969 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-03-22T05:34:18.596 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:18 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/2763944778' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:34:18.597 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:18 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/2763944778' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:34:18.621 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph health detail' 2024-03-22T05:34:18.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:18 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/2763944778' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:34:19.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:19 smithi005 ceph-mon[24793]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:34:19.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:19 smithi005 ceph-mon[29523]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:34:19.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:19 smithi186 ceph-mon[25720]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:34:20.737 INFO:teuthology.orchestra.run.smithi005.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-22T05:34:20.738 INFO:teuthology.orchestra.run.smithi005.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-22T05:34:20.738 INFO:teuthology.orchestra.run.smithi005.stdout: daemon grafana.a on smithi186 is in error state 2024-03-22T05:34:21.388 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph -s' 2024-03-22T05:34:21.699 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:21 smithi005 ceph-mon[24793]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:34:21.699 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:21 smithi005 ceph-mon[24793]: from='client.34937 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:34:21.699 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:21 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/1589084209' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T05:34:21.699 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:21 smithi005 ceph-mon[29523]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:34:21.699 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:21 smithi005 ceph-mon[29523]: from='client.34937 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:34:21.699 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:21 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/1589084209' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T05:34:21.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:21 smithi186 ceph-mon[25720]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:34:21.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:21 smithi186 ceph-mon[25720]: from='client.34937 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:34:21.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:21 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/1589084209' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T05:34:21.794 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:34:21 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:34:21] "GET /metrics HTTP/1.1" 200 37957 "" "Prometheus/2.33.4" 2024-03-22T05:34:23.483 INFO:teuthology.orchestra.run.smithi005.stdout: cluster: 2024-03-22T05:34:23.483 INFO:teuthology.orchestra.run.smithi005.stdout: id: f59f36e2-e80a-11ee-95cd-87774f69a715 2024-03-22T05:34:23.483 INFO:teuthology.orchestra.run.smithi005.stdout: health: HEALTH_WARN 2024-03-22T05:34:23.483 INFO:teuthology.orchestra.run.smithi005.stdout: 1 failed cephadm daemon(s) 2024-03-22T05:34:23.483 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-22T05:34:23.483 INFO:teuthology.orchestra.run.smithi005.stdout: services: 2024-03-22T05:34:23.483 INFO:teuthology.orchestra.run.smithi005.stdout: mon: 3 daemons, quorum a,c,b (age 18m) 2024-03-22T05:34:23.483 INFO:teuthology.orchestra.run.smithi005.stdout: mgr: x(active, since 6m), standbys: y 2024-03-22T05:34:23.484 INFO:teuthology.orchestra.run.smithi005.stdout: osd: 8 osds: 8 up (since 15m), 8 in (since 15m) 2024-03-22T05:34:23.484 INFO:teuthology.orchestra.run.smithi005.stdout: rgw: 2 daemons active (2 hosts, 1 zones) 2024-03-22T05:34:23.484 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-22T05:34:23.484 INFO:teuthology.orchestra.run.smithi005.stdout: data: 2024-03-22T05:34:23.484 INFO:teuthology.orchestra.run.smithi005.stdout: pools: 6 pools, 161 pgs 2024-03-22T05:34:23.484 INFO:teuthology.orchestra.run.smithi005.stdout: objects: 241 objects, 585 KiB 2024-03-22T05:34:23.484 INFO:teuthology.orchestra.run.smithi005.stdout: usage: 95 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:34:23.484 INFO:teuthology.orchestra.run.smithi005.stdout: pgs: 161 active+clean 2024-03-22T05:34:23.484 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-22T05:34:23.484 INFO:teuthology.orchestra.run.smithi005.stdout: io: 2024-03-22T05:34:23.484 INFO:teuthology.orchestra.run.smithi005.stdout: client: 1.2 KiB/s rd, 1 op/s rd, 0 op/s wr 2024-03-22T05:34:23.484 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-22T05:34:23.784 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:23 smithi005 ceph-mon[24793]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:34:23.784 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:23 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/4190277800' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-22T05:34:23.784 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:23 smithi005 ceph-mon[29523]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:34:23.784 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:23 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/4190277800' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-22T05:34:23.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:23 smithi186 ceph-mon[25720]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:34:23.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:23 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/4190277800' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-22T05:34:24.112 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph mgr fail' 2024-03-22T05:34:25.679 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:25 smithi005 ceph-mon[24793]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:34:25.680 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:25 smithi005 ceph-mon[29523]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:34:25.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:25 smithi186 ceph-mon[25720]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:34:26.630 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:26 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/603258955' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-03-22T05:34:26.630 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:26 smithi005 ceph-mon[24793]: osdmap e102: 8 total, 8 up, 8 in 2024-03-22T05:34:26.631 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:26 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: [22/Mar/2024:05:34:26] ENGINE Bus STOPPING 2024-03-22T05:34:26.631 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:26 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: [22/Mar/2024:05:34:26] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-22T05:34:26.631 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:26 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: [22/Mar/2024:05:34:26] ENGINE Bus STOPPED 2024-03-22T05:34:26.631 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:26 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/603258955' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-03-22T05:34:26.631 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:26 smithi005 ceph-mon[29523]: osdmap e102: 8 total, 8 up, 8 in 2024-03-22T05:34:26.777 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:26 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/603258955' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-03-22T05:34:26.777 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:26 smithi186 ceph-mon[25720]: osdmap e102: 8 total, 8 up, 8 in 2024-03-22T05:34:26.777 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:34:26 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: 2024-03-22T05:34:26.530+0000 7f0400ed5700 -1 mgr handle_mgr_map I was active but no longer am 2024-03-22T05:34:26.777 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:34:26 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: ignoring --setuser ceph since I am not root 2024-03-22T05:34:26.777 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:34:26 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: ignoring --setgroup ceph since I am not root 2024-03-22T05:34:26.777 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:34:26 smithi186 ceph-mgr[45898]: -- 172.21.15.186:0/1669812911 <== mon.2 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5573d89885a0 con 0x5573d895cc00 2024-03-22T05:34:26.916 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:26 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: [22/Mar/2024:05:34:26] ENGINE Bus STARTING 2024-03-22T05:34:26.916 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:26 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: [22/Mar/2024:05:34:26] ENGINE Serving on http://:::9283 2024-03-22T05:34:26.917 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:26 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: [22/Mar/2024:05:34:26] ENGINE Bus STARTED 2024-03-22T05:34:27.224 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'sleep 180' 2024-03-22T05:34:27.526 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:34:27 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: 2024-03-22T05:34:27.120+0000 7efc35346200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-22T05:34:27.527 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:34:27 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: 2024-03-22T05:34:27.217+0000 7efc35346200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-22T05:34:27.786 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:27 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/603258955' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-03-22T05:34:27.786 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:27 smithi005 ceph-mon[24793]: mgrmap e29: y(active, starting, since 0.353326s) 2024-03-22T05:34:27.787 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:27 smithi005 ceph-mon[24793]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-22T05:34:27.787 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:27 smithi005 ceph-mon[24793]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T05:34:27.787 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:27 smithi005 ceph-mon[24793]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T05:34:27.787 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:27 smithi005 ceph-mon[24793]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-22T05:34:27.787 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:27 smithi005 ceph-mon[24793]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T05:34:27.787 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:27 smithi005 ceph-mon[24793]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T05:34:27.787 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:27 smithi005 ceph-mon[24793]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T05:34:27.787 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:27 smithi005 ceph-mon[24793]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T05:34:27.787 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:27 smithi005 ceph-mon[24793]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T05:34:27.787 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:27 smithi005 ceph-mon[24793]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T05:34:27.787 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:27 smithi005 ceph-mon[24793]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-22T05:34:27.787 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:27 smithi005 ceph-mon[24793]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-22T05:34:27.787 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:27 smithi005 ceph-mon[24793]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-22T05:34:27.788 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:27 smithi005 ceph-mon[24793]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-22T05:34:27.788 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:27 smithi005 ceph-mon[24793]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-22T05:34:27.788 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:27 smithi005 ceph-mon[24793]: Manager daemon y is now available 2024-03-22T05:34:27.788 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:27 smithi005 ceph-mon[24793]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:34:27.788 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:27 smithi005 ceph-mon[24793]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:34:27.788 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:27 smithi005 ceph-mon[24793]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-22T05:34:27.788 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:27 smithi005 ceph-mon[24793]: from='mgr.25111 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-22T05:34:27.788 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:27 smithi005 ceph-mon[24793]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-22T05:34:27.788 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:27 smithi005 ceph-mon[24793]: from='mgr.25111 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-22T05:34:27.788 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:27 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/603258955' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-03-22T05:34:27.788 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:27 smithi005 ceph-mon[29523]: mgrmap e29: y(active, starting, since 0.353326s) 2024-03-22T05:34:27.788 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:27 smithi005 ceph-mon[29523]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-22T05:34:27.788 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:27 smithi005 ceph-mon[29523]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T05:34:27.789 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:27 smithi005 ceph-mon[29523]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T05:34:27.789 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:27 smithi005 ceph-mon[29523]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-22T05:34:27.789 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:27 smithi005 ceph-mon[29523]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T05:34:27.789 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:27 smithi005 ceph-mon[29523]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T05:34:27.789 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:27 smithi005 ceph-mon[29523]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T05:34:27.789 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:27 smithi005 ceph-mon[29523]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T05:34:27.789 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:27 smithi005 ceph-mon[29523]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T05:34:27.789 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:27 smithi005 ceph-mon[29523]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T05:34:27.789 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:27 smithi005 ceph-mon[29523]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-22T05:34:27.789 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:27 smithi005 ceph-mon[29523]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-22T05:34:27.789 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:27 smithi005 ceph-mon[29523]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-22T05:34:27.789 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:27 smithi005 ceph-mon[29523]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-22T05:34:27.789 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:27 smithi005 ceph-mon[29523]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-22T05:34:27.789 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:27 smithi005 ceph-mon[29523]: Manager daemon y is now available 2024-03-22T05:34:27.790 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:27 smithi005 ceph-mon[29523]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:34:27.790 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:27 smithi005 ceph-mon[29523]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:34:27.790 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:27 smithi005 ceph-mon[29523]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-22T05:34:27.790 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:27 smithi005 ceph-mon[29523]: from='mgr.25111 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-22T05:34:27.790 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:27 smithi005 ceph-mon[29523]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-22T05:34:27.790 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:27 smithi005 ceph-mon[29523]: from='mgr.25111 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-22T05:34:27.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:27 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/603258955' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-03-22T05:34:27.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:27 smithi186 ceph-mon[25720]: mgrmap e29: y(active, starting, since 0.353326s) 2024-03-22T05:34:27.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:27 smithi186 ceph-mon[25720]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-22T05:34:27.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:27 smithi186 ceph-mon[25720]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T05:34:27.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:27 smithi186 ceph-mon[25720]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T05:34:27.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:27 smithi186 ceph-mon[25720]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-22T05:34:27.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:27 smithi186 ceph-mon[25720]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T05:34:27.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:27 smithi186 ceph-mon[25720]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T05:34:27.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:27 smithi186 ceph-mon[25720]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T05:34:27.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:27 smithi186 ceph-mon[25720]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T05:34:27.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:27 smithi186 ceph-mon[25720]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T05:34:27.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:27 smithi186 ceph-mon[25720]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T05:34:27.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:27 smithi186 ceph-mon[25720]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-22T05:34:27.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:27 smithi186 ceph-mon[25720]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-22T05:34:27.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:27 smithi186 ceph-mon[25720]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-22T05:34:27.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:27 smithi186 ceph-mon[25720]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-22T05:34:27.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:27 smithi186 ceph-mon[25720]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-22T05:34:27.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:27 smithi186 ceph-mon[25720]: Manager daemon y is now available 2024-03-22T05:34:27.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:27 smithi186 ceph-mon[25720]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:34:27.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:27 smithi186 ceph-mon[25720]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:34:27.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:27 smithi186 ceph-mon[25720]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-22T05:34:27.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:27 smithi186 ceph-mon[25720]: from='mgr.25111 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-22T05:34:27.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:27 smithi186 ceph-mon[25720]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-22T05:34:27.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:27 smithi186 ceph-mon[25720]: from='mgr.25111 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-22T05:34:28.139 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:34:27 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: 2024-03-22T05:34:27.902+0000 7efc35346200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-22T05:34:28.140 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:34:27 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: 2024-03-22T05:34:27.996+0000 7efc35346200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-22T05:34:28.140 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:34:28 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: 2024-03-22T05:34:28.140+0000 7efc35346200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-22T05:34:28.438 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:34:28 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: 2024-03-22T05:34:28.252+0000 7efc35346200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-22T05:34:28.439 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:34:28 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: 2024-03-22T05:34:28.340+0000 7efc35346200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-22T05:34:28.733 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:28 smithi186 ceph-mon[25720]: [22/Mar/2024:05:34:27] ENGINE Bus STARTING 2024-03-22T05:34:28.733 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:28 smithi186 ceph-mon[25720]: [22/Mar/2024:05:34:27] ENGINE Serving on https://172.21.15.5:7150 2024-03-22T05:34:28.733 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:28 smithi186 ceph-mon[25720]: mgrmap e30: y(active, since 1.35998s) 2024-03-22T05:34:28.733 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:28 smithi186 ceph-mon[25720]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:34:28.733 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:28 smithi186 ceph-mon[25720]: [22/Mar/2024:05:34:27] ENGINE Serving on http://172.21.15.5:8765 2024-03-22T05:34:28.733 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:28 smithi186 ceph-mon[25720]: [22/Mar/2024:05:34:27] ENGINE Bus STARTED 2024-03-22T05:34:28.733 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:34:28 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: 2024-03-22T05:34:28.619+0000 7efc35346200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-22T05:34:28.734 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:34:28 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: 2024-03-22T05:34:28.733+0000 7efc35346200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-22T05:34:28.780 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:28 smithi005 ceph-mon[29523]: [22/Mar/2024:05:34:27] ENGINE Bus STARTING 2024-03-22T05:34:28.780 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:28 smithi005 ceph-mon[29523]: [22/Mar/2024:05:34:27] ENGINE Serving on https://172.21.15.5:7150 2024-03-22T05:34:28.780 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:28 smithi005 ceph-mon[29523]: mgrmap e30: y(active, since 1.35998s) 2024-03-22T05:34:28.780 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:28 smithi005 ceph-mon[29523]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:34:28.780 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:28 smithi005 ceph-mon[29523]: [22/Mar/2024:05:34:27] ENGINE Serving on http://172.21.15.5:8765 2024-03-22T05:34:28.780 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:28 smithi005 ceph-mon[29523]: [22/Mar/2024:05:34:27] ENGINE Bus STARTED 2024-03-22T05:34:28.780 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:28 smithi005 ceph-mon[24793]: [22/Mar/2024:05:34:27] ENGINE Bus STARTING 2024-03-22T05:34:28.780 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:28 smithi005 ceph-mon[24793]: [22/Mar/2024:05:34:27] ENGINE Serving on https://172.21.15.5:7150 2024-03-22T05:34:28.780 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:28 smithi005 ceph-mon[24793]: mgrmap e30: y(active, since 1.35998s) 2024-03-22T05:34:28.781 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:28 smithi005 ceph-mon[24793]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:34:28.781 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:28 smithi005 ceph-mon[24793]: [22/Mar/2024:05:34:27] ENGINE Serving on http://172.21.15.5:8765 2024-03-22T05:34:28.781 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:28 smithi005 ceph-mon[24793]: [22/Mar/2024:05:34:27] ENGINE Bus STARTED 2024-03-22T05:34:29.043 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:34:28 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: 2024-03-22T05:34:28.821+0000 7efc35346200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-22T05:34:29.043 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:34:28 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: 2024-03-22T05:34:28.933+0000 7efc35346200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-22T05:34:29.508 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:34:29 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: 2024-03-22T05:34:29.141+0000 7efc35346200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-22T05:34:29.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:29 smithi186 ceph-mon[25720]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-22T05:34:29.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:29 smithi186 ceph-mon[25720]: Cluster is now healthy 2024-03-22T05:34:29.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:29 smithi186 ceph-mon[25720]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:34:29.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:29 smithi186 ceph-mon[25720]: mgrmap e31: y(active, since 2s) 2024-03-22T05:34:29.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:29 smithi005 ceph-mon[29523]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-22T05:34:29.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:29 smithi005 ceph-mon[29523]: Cluster is now healthy 2024-03-22T05:34:29.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:29 smithi005 ceph-mon[29523]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:34:29.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:29 smithi005 ceph-mon[29523]: mgrmap e31: y(active, since 2s) 2024-03-22T05:34:29.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:29 smithi005 ceph-mon[24793]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-22T05:34:29.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:29 smithi005 ceph-mon[24793]: Cluster is now healthy 2024-03-22T05:34:29.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:29 smithi005 ceph-mon[24793]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:34:29.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:29 smithi005 ceph-mon[24793]: mgrmap e31: y(active, since 2s) 2024-03-22T05:34:30.318 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:34:30 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: 2024-03-22T05:34:30.160+0000 7efc35346200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-22T05:34:30.319 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:34:30 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: 2024-03-22T05:34:30.294+0000 7efc35346200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-22T05:34:30.983 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:34:30 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: 2024-03-22T05:34:30.651+0000 7efc35346200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-22T05:34:30.983 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:34:30 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: 2024-03-22T05:34:30.766+0000 7efc35346200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-22T05:34:30.983 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:34:30 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: 2024-03-22T05:34:30.983+0000 7efc35346200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-22T05:34:31.293 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:34:31 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: 2024-03-22T05:34:31.186+0000 7efc35346200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-22T05:34:31.770 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:31 smithi186 ceph-mon[25720]: from='client.34937 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:34:31.770 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:31 smithi186 ceph-mon[25720]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:34:31.770 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:31 smithi186 ceph-mon[25720]: mgrmap e32: y(active, since 4s) 2024-03-22T05:34:31.770 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:31 smithi186 ceph-mon[25720]: from='mgr.25111 ' entity='mgr.y' 2024-03-22T05:34:31.770 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:31 smithi186 ceph-mon[25720]: from='mgr.25111 ' entity='mgr.y' 2024-03-22T05:34:31.771 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:34:31 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: 2024-03-22T05:34:31.494+0000 7efc35346200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-22T05:34:31.771 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:34:31 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: 2024-03-22T05:34:31.770+0000 7efc35346200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-22T05:34:31.822 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:31 smithi005 ceph-mon[24793]: from='client.34937 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:34:31.822 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:31 smithi005 ceph-mon[24793]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:34:31.822 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:31 smithi005 ceph-mon[24793]: mgrmap e32: y(active, since 4s) 2024-03-22T05:34:31.822 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:31 smithi005 ceph-mon[24793]: from='mgr.25111 ' entity='mgr.y' 2024-03-22T05:34:31.822 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:31 smithi005 ceph-mon[24793]: from='mgr.25111 ' entity='mgr.y' 2024-03-22T05:34:31.823 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:31 smithi005 ceph-mon[29523]: from='client.34937 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:34:31.823 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:31 smithi005 ceph-mon[29523]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:34:31.823 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:31 smithi005 ceph-mon[29523]: mgrmap e32: y(active, since 4s) 2024-03-22T05:34:31.823 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:31 smithi005 ceph-mon[29523]: from='mgr.25111 ' entity='mgr.y' 2024-03-22T05:34:31.823 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:31 smithi005 ceph-mon[29523]: from='mgr.25111 ' entity='mgr.y' 2024-03-22T05:34:32.293 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:34:32 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: 2024-03-22T05:34:32.026+0000 7efc35346200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-22T05:34:33.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:33 smithi005 ceph-mon[24793]: from='mgr.25111 ' entity='mgr.y' 2024-03-22T05:34:33.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:33 smithi005 ceph-mon[24793]: from='mgr.25111 ' entity='mgr.y' 2024-03-22T05:34:33.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:33 smithi005 ceph-mon[24793]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:34:33.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:33 smithi005 ceph-mon[29523]: from='mgr.25111 ' entity='mgr.y' 2024-03-22T05:34:33.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:33 smithi005 ceph-mon[29523]: from='mgr.25111 ' entity='mgr.y' 2024-03-22T05:34:33.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:33 smithi005 ceph-mon[29523]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:34:33.293 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:34:32 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: 2024-03-22T05:34:32.841+0000 7efc35346200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-22T05:34:33.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:33 smithi186 ceph-mon[25720]: from='mgr.25111 ' entity='mgr.y' 2024-03-22T05:34:33.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:33 smithi186 ceph-mon[25720]: from='mgr.25111 ' entity='mgr.y' 2024-03-22T05:34:33.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:33 smithi186 ceph-mon[25720]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:34:33.747 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:34:33 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: 2024-03-22T05:34:33.423+0000 7efc35346200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-22T05:34:33.747 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:34:33 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: 2024-03-22T05:34:33.523+0000 7efc35346200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-22T05:34:34.044 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:34:33 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: 2024-03-22T05:34:33.748+0000 7efc35346200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-22T05:34:34.044 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:34:33 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: [22/Mar/2024:05:34:33] ENGINE Bus STARTING 2024-03-22T05:34:34.044 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:34:33 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: CherryPy Checker: 2024-03-22T05:34:34.044 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:34:33 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: The Application mounted at '' has an empty config. 2024-03-22T05:34:34.044 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:34:33 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: 2024-03-22T05:34:34.044 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:34:33 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: [22/Mar/2024:05:34:33] ENGINE Serving on http://:::9283 2024-03-22T05:34:34.044 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:34:33 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: [22/Mar/2024:05:34:33] ENGINE Bus STARTED 2024-03-22T05:34:34.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:34 smithi005 ceph-mon[24793]: from='mgr.25111 ' entity='mgr.y' 2024-03-22T05:34:34.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:34 smithi005 ceph-mon[24793]: from='mgr.25111 ' entity='mgr.y' 2024-03-22T05:34:34.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:34 smithi005 ceph-mon[24793]: Standby manager daemon x started 2024-03-22T05:34:34.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:34 smithi005 ceph-mon[24793]: from='mgr.? 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-22T05:34:34.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:34 smithi005 ceph-mon[24793]: from='mgr.? 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-22T05:34:34.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:34 smithi005 ceph-mon[24793]: from='mgr.? 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-22T05:34:34.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:34 smithi005 ceph-mon[24793]: from='mgr.? 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-22T05:34:34.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:34 smithi005 ceph-mon[29523]: from='mgr.25111 ' entity='mgr.y' 2024-03-22T05:34:34.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:34 smithi005 ceph-mon[29523]: from='mgr.25111 ' entity='mgr.y' 2024-03-22T05:34:34.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:34 smithi005 ceph-mon[29523]: Standby manager daemon x started 2024-03-22T05:34:34.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:34 smithi005 ceph-mon[29523]: from='mgr.? 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-22T05:34:34.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:34 smithi005 ceph-mon[29523]: from='mgr.? 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-22T05:34:34.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:34 smithi005 ceph-mon[29523]: from='mgr.? 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-22T05:34:34.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:34 smithi005 ceph-mon[29523]: from='mgr.? 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-22T05:34:34.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:34 smithi186 ceph-mon[25720]: from='mgr.25111 ' entity='mgr.y' 2024-03-22T05:34:34.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:34 smithi186 ceph-mon[25720]: from='mgr.25111 ' entity='mgr.y' 2024-03-22T05:34:34.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:34 smithi186 ceph-mon[25720]: Standby manager daemon x started 2024-03-22T05:34:34.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:34 smithi186 ceph-mon[25720]: from='mgr.? 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-22T05:34:34.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:34 smithi186 ceph-mon[25720]: from='mgr.? 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-22T05:34:34.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:34 smithi186 ceph-mon[25720]: from='mgr.? 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-22T05:34:34.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:34 smithi186 ceph-mon[25720]: from='mgr.? 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-22T05:34:35.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:35 smithi186 ceph-mon[25720]: from='mgr.25111 ' entity='mgr.y' 2024-03-22T05:34:35.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:35 smithi186 ceph-mon[25720]: from='mgr.25111 ' entity='mgr.y' 2024-03-22T05:34:35.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:35 smithi186 ceph-mon[25720]: mgrmap e33: y(active, since 8s), standbys: x 2024-03-22T05:34:35.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:35 smithi186 ceph-mon[25720]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-22T05:34:35.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:35 smithi186 ceph-mon[25720]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:34:35.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:35 smithi186 ceph-mon[25720]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:34:35.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:35 smithi186 ceph-mon[25720]: Updating smithi005:/etc/ceph/ceph.conf 2024-03-22T05:34:35.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:35 smithi186 ceph-mon[25720]: Updating smithi186:/etc/ceph/ceph.conf 2024-03-22T05:34:35.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:35 smithi186 ceph-mon[25720]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-22T05:34:35.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:35 smithi186 ceph-mon[25720]: Updating smithi005:/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T05:34:35.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:35 smithi186 ceph-mon[25720]: Updating smithi186:/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T05:34:35.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:35 smithi005 ceph-mon[29523]: from='mgr.25111 ' entity='mgr.y' 2024-03-22T05:34:35.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:35 smithi005 ceph-mon[29523]: from='mgr.25111 ' entity='mgr.y' 2024-03-22T05:34:35.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:35 smithi005 ceph-mon[29523]: mgrmap e33: y(active, since 8s), standbys: x 2024-03-22T05:34:35.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:35 smithi005 ceph-mon[29523]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-22T05:34:35.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:35 smithi005 ceph-mon[29523]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:34:35.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:35 smithi005 ceph-mon[29523]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:34:35.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:35 smithi005 ceph-mon[29523]: Updating smithi005:/etc/ceph/ceph.conf 2024-03-22T05:34:35.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:35 smithi005 ceph-mon[29523]: Updating smithi186:/etc/ceph/ceph.conf 2024-03-22T05:34:35.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:35 smithi005 ceph-mon[29523]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-22T05:34:35.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:35 smithi005 ceph-mon[29523]: Updating smithi005:/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T05:34:35.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:35 smithi005 ceph-mon[29523]: Updating smithi186:/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T05:34:35.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:35 smithi005 ceph-mon[24793]: from='mgr.25111 ' entity='mgr.y' 2024-03-22T05:34:35.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:35 smithi005 ceph-mon[24793]: from='mgr.25111 ' entity='mgr.y' 2024-03-22T05:34:35.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:35 smithi005 ceph-mon[24793]: mgrmap e33: y(active, since 8s), standbys: x 2024-03-22T05:34:35.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:35 smithi005 ceph-mon[24793]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-22T05:34:35.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:35 smithi005 ceph-mon[24793]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:34:35.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:35 smithi005 ceph-mon[24793]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:34:35.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:35 smithi005 ceph-mon[24793]: Updating smithi005:/etc/ceph/ceph.conf 2024-03-22T05:34:35.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:35 smithi005 ceph-mon[24793]: Updating smithi186:/etc/ceph/ceph.conf 2024-03-22T05:34:35.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:35 smithi005 ceph-mon[24793]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-22T05:34:35.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:35 smithi005 ceph-mon[24793]: Updating smithi005:/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T05:34:35.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:35 smithi005 ceph-mon[24793]: Updating smithi186:/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T05:34:36.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:36 smithi005 ceph-mon[24793]: Updating smithi005:/etc/ceph/ceph.client.admin.keyring 2024-03-22T05:34:36.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:36 smithi005 ceph-mon[24793]: Updating smithi186:/etc/ceph/ceph.client.admin.keyring 2024-03-22T05:34:36.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:36 smithi005 ceph-mon[24793]: Updating smithi005:/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-22T05:34:36.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:36 smithi005 ceph-mon[24793]: Updating smithi186:/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-22T05:34:36.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:36 smithi005 ceph-mon[24793]: from='mgr.25111 ' entity='mgr.y' 2024-03-22T05:34:36.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:36 smithi005 ceph-mon[24793]: from='mgr.25111 ' entity='mgr.y' 2024-03-22T05:34:36.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:36 smithi005 ceph-mon[24793]: from='mgr.25111 ' entity='mgr.y' 2024-03-22T05:34:36.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:36 smithi005 ceph-mon[24793]: from='mgr.25111 ' entity='mgr.y' 2024-03-22T05:34:36.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:36 smithi005 ceph-mon[24793]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 15 op/s 2024-03-22T05:34:36.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:36 smithi005 ceph-mon[24793]: from='mgr.25111 ' entity='mgr.y' 2024-03-22T05:34:36.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:36 smithi005 ceph-mon[24793]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:34:36.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:36 smithi005 ceph-mon[24793]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:34:36.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:36 smithi005 ceph-mon[24793]: from='mgr.25111 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:34:36.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:36 smithi005 ceph-mon[24793]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:34:36.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:36 smithi005 ceph-mon[24793]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:34:36.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:36 smithi005 ceph-mon[29523]: Updating smithi005:/etc/ceph/ceph.client.admin.keyring 2024-03-22T05:34:36.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:36 smithi005 ceph-mon[29523]: Updating smithi186:/etc/ceph/ceph.client.admin.keyring 2024-03-22T05:34:36.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:36 smithi005 ceph-mon[29523]: Updating smithi005:/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-22T05:34:36.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:36 smithi005 ceph-mon[29523]: Updating smithi186:/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-22T05:34:36.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:36 smithi005 ceph-mon[29523]: from='mgr.25111 ' entity='mgr.y' 2024-03-22T05:34:36.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:36 smithi005 ceph-mon[29523]: from='mgr.25111 ' entity='mgr.y' 2024-03-22T05:34:36.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:36 smithi005 ceph-mon[29523]: from='mgr.25111 ' entity='mgr.y' 2024-03-22T05:34:36.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:36 smithi005 ceph-mon[29523]: from='mgr.25111 ' entity='mgr.y' 2024-03-22T05:34:36.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:36 smithi005 ceph-mon[29523]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 15 op/s 2024-03-22T05:34:36.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:36 smithi005 ceph-mon[29523]: from='mgr.25111 ' entity='mgr.y' 2024-03-22T05:34:36.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:36 smithi005 ceph-mon[29523]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:34:36.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:36 smithi005 ceph-mon[29523]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:34:36.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:36 smithi005 ceph-mon[29523]: from='mgr.25111 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:34:36.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:36 smithi005 ceph-mon[29523]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:34:36.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:36 smithi005 ceph-mon[29523]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:34:36.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:36 smithi186 ceph-mon[25720]: Updating smithi005:/etc/ceph/ceph.client.admin.keyring 2024-03-22T05:34:36.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:36 smithi186 ceph-mon[25720]: Updating smithi186:/etc/ceph/ceph.client.admin.keyring 2024-03-22T05:34:36.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:36 smithi186 ceph-mon[25720]: Updating smithi005:/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-22T05:34:36.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:36 smithi186 ceph-mon[25720]: Updating smithi186:/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-22T05:34:36.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:36 smithi186 ceph-mon[25720]: from='mgr.25111 ' entity='mgr.y' 2024-03-22T05:34:36.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:36 smithi186 ceph-mon[25720]: from='mgr.25111 ' entity='mgr.y' 2024-03-22T05:34:36.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:36 smithi186 ceph-mon[25720]: from='mgr.25111 ' entity='mgr.y' 2024-03-22T05:34:36.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:36 smithi186 ceph-mon[25720]: from='mgr.25111 ' entity='mgr.y' 2024-03-22T05:34:36.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:36 smithi186 ceph-mon[25720]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 15 op/s 2024-03-22T05:34:36.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:36 smithi186 ceph-mon[25720]: from='mgr.25111 ' entity='mgr.y' 2024-03-22T05:34:36.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:36 smithi186 ceph-mon[25720]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:34:36.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:36 smithi186 ceph-mon[25720]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:34:36.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:36 smithi186 ceph-mon[25720]: from='mgr.25111 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:34:36.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:36 smithi186 ceph-mon[25720]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:34:36.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:36 smithi186 ceph-mon[25720]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:34:37.636 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:37 smithi005 ceph-mon[29523]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-22T05:34:37.636 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:37 smithi005 ceph-mon[24793]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-22T05:34:37.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:37 smithi186 ceph-mon[25720]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-22T05:34:38.294 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:34:37 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:34:37.800Z caller=refresh.go:98 level=error component="discovery manager notify" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.186:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.186:8765: connect: connection refused" 2024-03-22T05:34:38.294 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:34:37 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:34:37.800Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.186:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.186:8765: connect: connection refused" 2024-03-22T05:34:38.294 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:34:37 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:34:37.802Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.186:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.186:8765: connect: connection refused" 2024-03-22T05:34:38.294 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:34:37 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:34:37.802Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.186:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.186:8765: connect: connection refused" 2024-03-22T05:34:38.294 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:34:37 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:34:37.802Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.186:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.186:8765: connect: connection refused" 2024-03-22T05:34:38.586 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:38 smithi005 ceph-mon[24793]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-22T05:34:38.587 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:38 smithi005 ceph-mon[29523]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-22T05:34:38.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:38 smithi186 ceph-mon[25720]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-22T05:34:38.952 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:34:38.688+0000 7f8428d6c700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:34:38.952 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:34:38.952 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:34:38.952 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:34:38.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:34:38.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:34:38.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:34:38.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:34:38.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:34:38.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:34:38.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:34:38.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:34:38.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:34:38.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:34:38.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:34:38.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:34:38.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:34:38.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:34:38.954 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:34:38.954 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:34:38.954 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:34:38.954 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:34:38.954 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:34:38.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:34:38.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:34:38.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:34:38.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:34:38.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:34:38.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:34:38.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return future.result(timeout) 2024-03-22T05:34:38.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:34:38.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.__get_result() 2024-03-22T05:34:38.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:34:38.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: raise self._exception 2024-03-22T05:34:38.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:34:38.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: config_blobs=daemon_spec.final_config, 2024-03-22T05:34:38.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:34:38.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:34:38.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:34:38.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:34:38.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:34:38.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:34:38.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:34:38.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:34:38.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:34:38.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:34:38.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:34:38.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:34:38.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:34:38.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:34:38.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:34:38.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:34:38.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:34:38.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:34:38.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:34:38.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:34:38.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:34:38.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:34:38.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:34:39.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:39 smithi005 ceph-mon[24793]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:34:39.702 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:34:39.702 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:34:39.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:34:39.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:34:39.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:34:39.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:34:39.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:34:39.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:34:39.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:34:39.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:34:39.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:34:39.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:34:39.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:34:39.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:34:39.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:34:39.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:34:39.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:34:39.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:34:39.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:34:39.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:34:39.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:34:39.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:34:39.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:34:39.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:34:39.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:34:39.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:34:39.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:34:39.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:34:39.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: return future.result(timeout) 2024-03-22T05:34:39.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:34:39.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.__get_result() 2024-03-22T05:34:39.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:34:39.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: raise self._exception 2024-03-22T05:34:39.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:34:39.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:34:39.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:34:39.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:34:39.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:34:39.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:34:39.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:34:39.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:34:39.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:34:39.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:34:39.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:34:39.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:34:39.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:34:39.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:34:39.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:34:39.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:34:39.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:34:39.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:34:39.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:34:39.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:34:39.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:34:39.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:34:39.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:34:39.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:34:39.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:34:39.706 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:39 smithi005 ceph-mon[24793]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-22T05:34:39.707 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:39 smithi005 ceph-mon[29523]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:34:39.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:34:39.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:34:39.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:34:39.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:34:39.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:34:39.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:34:39.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:34:39.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:34:39.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:34:39.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:34:39.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:34:39.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:34:39.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:34:39.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:34:39.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:34:39.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:34:39.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:34:39.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:34:39.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:34:39.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:34:39.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:34:39.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:34:39.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:34:39.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:34:39.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:34:39.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:34:39.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:34:39.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:34:39.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: return future.result(timeout) 2024-03-22T05:34:39.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:34:39.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.__get_result() 2024-03-22T05:34:39.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:34:39.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: raise self._exception 2024-03-22T05:34:39.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:34:39.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:34:39.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:34:39.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:34:39.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:34:39.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:34:39.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:34:39.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:34:39.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:34:39.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:34:39.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:34:39.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:34:39.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:34:39.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:34:39.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:34:39.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:34:39.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:34:39.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:34:39.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:34:39.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:34:39.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:34:39.711 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:34:39.711 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:34:39.711 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:34:39.711 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:34:39.711 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:39 smithi005 ceph-mon[29523]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-22T05:34:39.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:39 smithi186 ceph-mon[25720]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:34:39.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:34:39.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:34:39.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:34:39.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:34:39.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:34:39.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:34:39.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:34:39.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:34:39.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:34:39.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:34:39.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:34:39.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:34:39.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:34:39.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:34:39.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:34:39.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:34:39.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:34:39.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:34:39.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:34:39.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:34:39.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:34:39.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:34:39.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:34:39.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:34:39.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:34:39.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:34:39.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:34:39.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:34:39.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: return future.result(timeout) 2024-03-22T05:34:39.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:34:39.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.__get_result() 2024-03-22T05:34:39.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:34:39.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: raise self._exception 2024-03-22T05:34:39.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:34:39.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:34:39.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:34:39.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:34:39.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:34:39.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:34:39.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:34:39.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:34:39.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:34:39.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:34:39.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:34:39.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:34:39.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:34:39.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:34:39.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:34:39.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:34:39.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:34:39.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:34:39.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:34:39.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:34:39.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:34:39.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:34:39.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:34:39.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:34:39.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:34:39.798 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:39 smithi186 ceph-mon[25720]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-22T05:34:40.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:40 smithi005 ceph-mon[29523]: Reconfiguring daemon prometheus.a on smithi186 2024-03-22T05:34:40.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:40 smithi005 ceph-mon[29523]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-22T05:34:40.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:40 smithi005 ceph-mon[24793]: Reconfiguring daemon prometheus.a on smithi186 2024-03-22T05:34:40.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:40 smithi005 ceph-mon[24793]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-22T05:34:40.744 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:40 smithi186 ceph-mon[25720]: Reconfiguring daemon prometheus.a on smithi186 2024-03-22T05:34:40.744 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:40 smithi186 ceph-mon[25720]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-22T05:34:41.793 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:34:41 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:34:41] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-22T05:34:42.086 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:34:41 smithi186 systemd[1]: Stopping Ceph prometheus.a for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:34:42.087 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:34:41 smithi186 bash[54882]: Error: no container with name or ID "ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus.a" found: no such container 2024-03-22T05:34:42.371 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:34:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:34:42.087Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-22T05:34:42.371 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:34:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:34:42.087Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-03-22T05:34:42.371 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:34:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:34:42.087Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-03-22T05:34:42.371 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:34:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:34:42.087Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-03-22T05:34:42.371 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:34:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:34:42.087Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-03-22T05:34:42.372 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:34:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:34:42.087Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-03-22T05:34:42.372 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:34:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:34:42.087Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-22T05:34:42.372 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:34:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:34:42.087Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-03-22T05:34:42.372 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:34:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:34:42.087Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-03-22T05:34:42.372 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:34:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:34:42.088Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-03-22T05:34:42.372 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:34:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:34:42.088Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-03-22T05:34:42.372 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:34:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[49224]: ts=2024-03-22T05:34:42.088Z caller=main.go:1066 level=info msg="See you next time!" 2024-03-22T05:34:42.372 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:34:42 smithi186 podman[54895]: 2024-03-22 05:34:42.097812019 +0000 UTC m=+0.119579422 container died c3a69c1f7e79f68a37cb6b6c272106ec38f748a73e302b0caec7f651c4e5ad09 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-22T05:34:42.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:42 smithi005 ceph-mon[24793]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-22T05:34:42.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:42 smithi005 ceph-mon[24793]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:34:42.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:42 smithi005 ceph-mon[29523]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-22T05:34:42.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:42 smithi005 ceph-mon[29523]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:34:42.737 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:42 smithi186 ceph-mon[25720]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-22T05:34:42.737 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:42 smithi186 ceph-mon[25720]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:34:42.738 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:34:42 smithi186 podman[54895]: 2024-03-22 05:34:42.470477042 +0000 UTC m=+0.492244455 container cleanup c3a69c1f7e79f68a37cb6b6c272106ec38f748a73e302b0caec7f651c4e5ad09 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-22T05:34:42.738 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:34:42 smithi186 bash[54895]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a 2024-03-22T05:34:42.738 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:34:42 smithi186 bash[54921]: Error: no container with name or ID "ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus.a" found: no such container 2024-03-22T05:34:42.996 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:34:42 smithi186 podman[54909]: 2024-03-22 05:34:42.738289708 +0000 UTC m=+0.642796427 container remove c3a69c1f7e79f68a37cb6b6c272106ec38f748a73e302b0caec7f651c4e5ad09 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-22T05:34:42.996 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:34:42 smithi186 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-03-22T05:34:42.997 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:34:42 smithi186 systemd[1]: Stopped Ceph prometheus.a for f59f36e2-e80a-11ee-95cd-87774f69a715. 2024-03-22T05:34:42.997 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:34:42 smithi186 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@prometheus.a.service: Consumed 2.009s CPU time. 2024-03-22T05:34:42.997 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:34:42 smithi186 systemd[1]: Starting Ceph prometheus.a for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:34:42.997 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:34:42 smithi186 podman[54994]: 2024-03-22 05:34:42.884561312 +0000 UTC m=+0.019507818 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-03-22T05:34:42.997 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:34:42 smithi186 podman[54994]: 2024-03-22 05:34:42.997596497 +0000 UTC m=+0.132543029 container create 546020b39d245819eb217e9db65507e56522d1ae4801d13358ab95ee5127c25b (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-22T05:34:43.294 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:34:43 smithi186 podman[54994]: 2024-03-22 05:34:43.199773251 +0000 UTC m=+0.334719752 container init 546020b39d245819eb217e9db65507e56522d1ae4801d13358ab95ee5127c25b (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-22T05:34:43.294 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:34:43 smithi186 podman[54994]: 2024-03-22 05:34:43.202172197 +0000 UTC m=+0.337118684 container start 546020b39d245819eb217e9db65507e56522d1ae4801d13358ab95ee5127c25b (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-22T05:34:43.294 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:34:43 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[55010]: ts=2024-03-22T05:34:43.226Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-03-22T05:34:43.294 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:34:43 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[55010]: ts=2024-03-22T05:34:43.227Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-03-22T05:34:43.294 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:34:43 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[55010]: ts=2024-03-22T05:34:43.227Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-03-22T05:34:43.294 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:34:43 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[55010]: ts=2024-03-22T05:34:43.227Z caller=main.go:518 level=info host_details="(Linux 5.14.0-430.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024 x86_64 smithi186 (none))" 2024-03-22T05:34:43.294 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:34:43 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[55010]: ts=2024-03-22T05:34:43.227Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-22T05:34:43.294 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:34:43 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[55010]: ts=2024-03-22T05:34:43.227Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-22T05:34:43.295 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:34:43 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[55010]: ts=2024-03-22T05:34:43.228Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-03-22T05:34:43.295 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:34:43 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[55010]: ts=2024-03-22T05:34:43.228Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-03-22T05:34:43.295 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:34:43 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[55010]: ts=2024-03-22T05:34:43.229Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-03-22T05:34:43.295 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:34:43 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[55010]: ts=2024-03-22T05:34:43.231Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-22T05:34:43.295 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:34:43 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[55010]: ts=2024-03-22T05:34:43.231Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.513µs 2024-03-22T05:34:43.295 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:34:43 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[55010]: ts=2024-03-22T05:34:43.231Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-22T05:34:43.662 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:34:43 smithi186 bash[54994]: 546020b39d245819eb217e9db65507e56522d1ae4801d13358ab95ee5127c25b 2024-03-22T05:34:43.662 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:34:43 smithi186 systemd[1]: Started Ceph prometheus.a for f59f36e2-e80a-11ee-95cd-87774f69a715. 2024-03-22T05:34:43.662 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:34:43 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[55010]: ts=2024-03-22T05:34:43.434Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=3 2024-03-22T05:34:44.043 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:34:43 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[55010]: ts=2024-03-22T05:34:43.663Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=3 2024-03-22T05:34:44.044 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:34:43 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[55010]: ts=2024-03-22T05:34:43.873Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=3 2024-03-22T05:34:44.044 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:34:43 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[55010]: ts=2024-03-22T05:34:43.874Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=3 2024-03-22T05:34:44.044 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:34:43 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[55010]: ts=2024-03-22T05:34:43.874Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=55.393µs wal_replay_duration=642.685419ms total_replay_duration=642.756651ms 2024-03-22T05:34:44.044 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:34:43 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[55010]: ts=2024-03-22T05:34:43.878Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-03-22T05:34:44.044 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:34:43 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[55010]: ts=2024-03-22T05:34:43.878Z caller=main.go:947 level=info msg="TSDB started" 2024-03-22T05:34:44.044 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:34:43 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[55010]: ts=2024-03-22T05:34:43.878Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-22T05:34:44.044 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:34:43 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[55010]: ts=2024-03-22T05:34:43.909Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=31.14208ms db_storage=877ns remote_storage=1.462µs web_handler=477ns query_engine=771ns scrape=662.446µs scrape_sd=145.829µs notify=23.412µs notify_sd=70.289µs rules=29.723ms 2024-03-22T05:34:44.044 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:34:43 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[55010]: ts=2024-03-22T05:34:43.909Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-03-22T05:34:44.446 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:44 smithi186 ceph-mon[25720]: from='mgr.25111 ' entity='mgr.y' 2024-03-22T05:34:44.446 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:44 smithi186 ceph-mon[25720]: from='mgr.25111 ' entity='mgr.y' 2024-03-22T05:34:44.447 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:44 smithi186 ceph-mon[25720]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-22T05:34:44.447 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:44 smithi186 ceph-mon[25720]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-22T05:34:44.447 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:44 smithi186 ceph-mon[25720]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:34:44.447 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:44 smithi186 ceph-mon[25720]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-22T05:34:44.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:44 smithi005 ceph-mon[24793]: from='mgr.25111 ' entity='mgr.y' 2024-03-22T05:34:44.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:44 smithi005 ceph-mon[24793]: from='mgr.25111 ' entity='mgr.y' 2024-03-22T05:34:44.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:44 smithi005 ceph-mon[24793]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-22T05:34:44.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:44 smithi005 ceph-mon[24793]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-22T05:34:44.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:44 smithi005 ceph-mon[24793]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:34:44.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:44 smithi005 ceph-mon[24793]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-22T05:34:44.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:44 smithi005 ceph-mon[29523]: from='mgr.25111 ' entity='mgr.y' 2024-03-22T05:34:44.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:44 smithi005 ceph-mon[29523]: from='mgr.25111 ' entity='mgr.y' 2024-03-22T05:34:44.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:44 smithi005 ceph-mon[29523]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-22T05:34:44.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:44 smithi005 ceph-mon[29523]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-22T05:34:44.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:44 smithi005 ceph-mon[29523]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:34:44.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:44 smithi005 ceph-mon[29523]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-22T05:34:46.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:46 smithi005 ceph-mon[24793]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 471 B/s rd, 0 op/s 2024-03-22T05:34:46.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:46 smithi005 ceph-mon[29523]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 471 B/s rd, 0 op/s 2024-03-22T05:34:46.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:46 smithi186 ceph-mon[25720]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 471 B/s rd, 0 op/s 2024-03-22T05:34:48.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:48 smithi005 ceph-mon[24793]: from='mgr.25111 ' entity='mgr.y' 2024-03-22T05:34:48.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:48 smithi005 ceph-mon[24793]: from='mgr.25111 ' entity='mgr.y' 2024-03-22T05:34:48.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:48 smithi005 ceph-mon[24793]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:34:48.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:48 smithi005 ceph-mon[24793]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:34:48.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:48 smithi005 ceph-mon[24793]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 436 B/s rd, 0 op/s 2024-03-22T05:34:48.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:48 smithi005 ceph-mon[24793]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 526 B/s rd, 0 op/s 2024-03-22T05:34:48.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:48 smithi005 ceph-mon[24793]: from='mgr.25111 ' entity='mgr.y' 2024-03-22T05:34:48.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:48 smithi005 ceph-mon[24793]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:34:48.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:48 smithi005 ceph-mon[24793]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:34:48.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:48 smithi005 ceph-mon[24793]: from='mgr.25111 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:34:48.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:48 smithi005 ceph-mon[24793]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:34:48.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:48 smithi005 ceph-mon[24793]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:34:48.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:48 smithi005 ceph-mon[29523]: from='mgr.25111 ' entity='mgr.y' 2024-03-22T05:34:48.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:48 smithi005 ceph-mon[29523]: from='mgr.25111 ' entity='mgr.y' 2024-03-22T05:34:48.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:48 smithi005 ceph-mon[29523]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:34:48.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:48 smithi005 ceph-mon[29523]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:34:48.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:48 smithi005 ceph-mon[29523]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 436 B/s rd, 0 op/s 2024-03-22T05:34:48.204 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:48 smithi005 ceph-mon[29523]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 526 B/s rd, 0 op/s 2024-03-22T05:34:48.204 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:48 smithi005 ceph-mon[29523]: from='mgr.25111 ' entity='mgr.y' 2024-03-22T05:34:48.204 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:48 smithi005 ceph-mon[29523]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:34:48.204 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:48 smithi005 ceph-mon[29523]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:34:48.204 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:48 smithi005 ceph-mon[29523]: from='mgr.25111 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:34:48.204 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:48 smithi005 ceph-mon[29523]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:34:48.204 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:48 smithi005 ceph-mon[29523]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:34:48.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:48 smithi186 ceph-mon[25720]: from='mgr.25111 ' entity='mgr.y' 2024-03-22T05:34:48.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:48 smithi186 ceph-mon[25720]: from='mgr.25111 ' entity='mgr.y' 2024-03-22T05:34:48.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:48 smithi186 ceph-mon[25720]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:34:48.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:48 smithi186 ceph-mon[25720]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:34:48.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:48 smithi186 ceph-mon[25720]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 436 B/s rd, 0 op/s 2024-03-22T05:34:48.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:48 smithi186 ceph-mon[25720]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 526 B/s rd, 0 op/s 2024-03-22T05:34:48.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:48 smithi186 ceph-mon[25720]: from='mgr.25111 ' entity='mgr.y' 2024-03-22T05:34:48.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:48 smithi186 ceph-mon[25720]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:34:48.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:48 smithi186 ceph-mon[25720]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:34:48.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:48 smithi186 ceph-mon[25720]: from='mgr.25111 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:34:48.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:48 smithi186 ceph-mon[25720]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:34:48.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:48 smithi186 ceph-mon[25720]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:34:50.468 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:50 smithi005 ceph-mon[29523]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:34:50.468 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:50 smithi005 ceph-mon[24793]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:34:50.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:50 smithi186 ceph-mon[25720]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:34:51.702 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:34:51.627+0000 7f8428d6c700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-iscsi-foo-smithi005-mcdwbp 2024-03-22T05:34:51.703 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: /usr/bin/podman: stderr Error: no such container ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-iscsi-foo-smithi005-mcdwbp 2024-03-22T05:34:51.703 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-iscsi.foo.smithi005.mcdwbp 2024-03-22T05:34:51.703 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: /usr/bin/podman: stderr Error: no such container ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-iscsi.foo.smithi005.mcdwbp 2024-03-22T05:34:51.703 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:34:51.703 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:34:51.703 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:34:51.703 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:34:51.703 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:34:51.703 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:34:51.703 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:34:51.703 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:34:51.703 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:34:51.703 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:34:51.704 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:34:51.704 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:34:51.704 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:34:51.704 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:34:51.704 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:34:51.704 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:34:51.705 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:34:51.705 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:34:51.705 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:34:51.705 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:34:51.705 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:34:51.705 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:34:51.705 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:34:51.705 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:34:51.705 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:34:51.705 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:34:51.705 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:34:51.705 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:34:51.705 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:34:51.705 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return future.result(timeout) 2024-03-22T05:34:51.706 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:34:51.706 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.__get_result() 2024-03-22T05:34:51.706 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:34:51.706 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: raise self._exception 2024-03-22T05:34:51.706 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:34:51.706 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: config_blobs=daemon_spec.final_config, 2024-03-22T05:34:51.706 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:34:51.706 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:34:51.706 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-iscsi-foo-smithi005-mcdwbp 2024-03-22T05:34:51.706 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: /usr/bin/podman: stderr Error: no such container ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-iscsi-foo-smithi005-mcdwbp 2024-03-22T05:34:51.706 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-iscsi.foo.smithi005.mcdwbp 2024-03-22T05:34:51.706 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: /usr/bin/podman: stderr Error: no such container ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-iscsi.foo.smithi005.mcdwbp 2024-03-22T05:34:51.706 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:34:51.706 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:34:51.707 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:34:51.707 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:34:51.707 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:34:51.707 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:34:51.707 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:34:51.707 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:34:51.707 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:34:51.707 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:34:51.707 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:34:51.707 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:34:51.707 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:34:51.707 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:34:51.707 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:34:51.707 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:34:51.707 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:34:51.708 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:34:51.708 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:34:51.708 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:34:51.708 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:34:52.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:52 smithi005 ceph-mon[29523]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:34:52.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:52 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/2855112162' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-22T05:34:52.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:52 smithi005 ceph-mon[29523]: cephadm exited with an error code: 1, stderr: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-iscsi-foo-smithi005-mcdwbp 2024-03-22T05:34:52.703 INFO:journalctl@ceph.mon.c.smithi005.stdout: /usr/bin/podman: stderr Error: no such container ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-iscsi-foo-smithi005-mcdwbp 2024-03-22T05:34:52.703 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-iscsi.foo.smithi005.mcdwbp 2024-03-22T05:34:52.703 INFO:journalctl@ceph.mon.c.smithi005.stdout: /usr/bin/podman: stderr Error: no such container ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-iscsi.foo.smithi005.mcdwbp 2024-03-22T05:34:52.703 INFO:journalctl@ceph.mon.c.smithi005.stdout: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:34:52.703 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:34:52.703 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:34:52.703 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:34:52.703 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:34:52.703 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:34:52.703 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:34:52.703 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:34:52.703 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:34:52.703 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:34:52.704 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:34:52.704 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:34:52.704 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:34:52.704 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:34:52.704 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:34:52.704 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:34:52.704 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:34:52.704 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:34:52.704 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:34:52.704 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:34:52.704 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:34:52.704 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:34:52.704 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:34:52.704 INFO:journalctl@ceph.mon.c.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:34:52.704 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:34:52.705 INFO:journalctl@ceph.mon.c.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:34:52.705 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:34:52.705 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:34:52.705 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:34:52.705 INFO:journalctl@ceph.mon.c.smithi005.stdout: return future.result(timeout) 2024-03-22T05:34:52.705 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:34:52.705 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.__get_result() 2024-03-22T05:34:52.705 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:34:52.705 INFO:journalctl@ceph.mon.c.smithi005.stdout: raise self._exception 2024-03-22T05:34:52.705 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:34:52.705 INFO:journalctl@ceph.mon.c.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:34:52.705 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:34:52.705 INFO:journalctl@ceph.mon.c.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:34:52.705 INFO:journalctl@ceph.mon.c.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-iscsi-foo-smithi005-mcdwbp 2024-03-22T05:34:52.705 INFO:journalctl@ceph.mon.c.smithi005.stdout: /usr/bin/podman: stderr Error: no such container ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-iscsi-foo-smithi005-mcdwbp 2024-03-22T05:34:52.706 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-iscsi.foo.smithi005.mcdwbp 2024-03-22T05:34:52.706 INFO:journalctl@ceph.mon.c.smithi005.stdout: /usr/bin/podman: stderr Error: no such container ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-iscsi.foo.smithi005.mcdwbp 2024-03-22T05:34:52.706 INFO:journalctl@ceph.mon.c.smithi005.stdout: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:34:52.706 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:34:52.706 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:34:52.706 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:34:52.706 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:34:52.706 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:34:52.706 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:34:52.706 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:34:52.706 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:34:52.706 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:34:52.706 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:34:52.706 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:34:52.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:34:52.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:34:52.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:34:52.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:34:52.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:34:52.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:34:52.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:34:52.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:34:52.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:34:52.707 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:52 smithi005 ceph-mon[24793]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:34:52.707 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:52 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/2855112162' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-22T05:34:52.708 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:52 smithi005 ceph-mon[24793]: cephadm exited with an error code: 1, stderr: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-iscsi-foo-smithi005-mcdwbp 2024-03-22T05:34:52.708 INFO:journalctl@ceph.mon.a.smithi005.stdout: /usr/bin/podman: stderr Error: no such container ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-iscsi-foo-smithi005-mcdwbp 2024-03-22T05:34:52.708 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-iscsi.foo.smithi005.mcdwbp 2024-03-22T05:34:52.708 INFO:journalctl@ceph.mon.a.smithi005.stdout: /usr/bin/podman: stderr Error: no such container ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-iscsi.foo.smithi005.mcdwbp 2024-03-22T05:34:52.708 INFO:journalctl@ceph.mon.a.smithi005.stdout: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:34:52.708 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:34:52.708 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:34:52.708 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:34:52.708 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:34:52.708 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:34:52.708 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:34:52.708 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:34:52.708 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:34:52.709 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:34:52.709 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:34:52.709 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:34:52.709 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:34:52.709 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:34:52.709 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:34:52.709 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:34:52.709 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:34:52.709 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:34:52.709 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:34:52.709 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:34:52.709 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:34:52.709 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:34:52.710 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:34:52.710 INFO:journalctl@ceph.mon.a.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:34:52.710 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:34:52.710 INFO:journalctl@ceph.mon.a.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:34:52.710 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:34:52.710 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:34:52.710 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:34:52.710 INFO:journalctl@ceph.mon.a.smithi005.stdout: return future.result(timeout) 2024-03-22T05:34:52.710 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:34:52.710 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.__get_result() 2024-03-22T05:34:52.710 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:34:52.711 INFO:journalctl@ceph.mon.a.smithi005.stdout: raise self._exception 2024-03-22T05:34:52.711 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:34:52.711 INFO:journalctl@ceph.mon.a.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:34:52.711 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:34:52.711 INFO:journalctl@ceph.mon.a.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:34:52.711 INFO:journalctl@ceph.mon.a.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-iscsi-foo-smithi005-mcdwbp 2024-03-22T05:34:52.711 INFO:journalctl@ceph.mon.a.smithi005.stdout: /usr/bin/podman: stderr Error: no such container ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-iscsi-foo-smithi005-mcdwbp 2024-03-22T05:34:52.711 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-iscsi.foo.smithi005.mcdwbp 2024-03-22T05:34:52.711 INFO:journalctl@ceph.mon.a.smithi005.stdout: /usr/bin/podman: stderr Error: no such container ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-iscsi.foo.smithi005.mcdwbp 2024-03-22T05:34:52.711 INFO:journalctl@ceph.mon.a.smithi005.stdout: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:34:52.711 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:34:52.712 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:34:52.712 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:34:52.712 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:34:52.712 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:34:52.712 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:34:52.712 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:34:52.712 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:34:52.712 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:34:52.712 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:34:52.712 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:34:52.712 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:34:52.712 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:34:52.712 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:34:52.712 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:34:52.713 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:34:52.713 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:34:52.713 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:34:52.713 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:34:52.713 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:34:52.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:52 smithi186 ceph-mon[25720]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:34:52.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:52 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/2855112162' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-22T05:34:52.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:52 smithi186 ceph-mon[25720]: cephadm exited with an error code: 1, stderr: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-iscsi-foo-smithi005-mcdwbp 2024-03-22T05:34:52.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: /usr/bin/podman: stderr Error: no such container ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-iscsi-foo-smithi005-mcdwbp 2024-03-22T05:34:52.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-iscsi.foo.smithi005.mcdwbp 2024-03-22T05:34:52.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: /usr/bin/podman: stderr Error: no such container ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-iscsi.foo.smithi005.mcdwbp 2024-03-22T05:34:52.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:34:52.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:34:52.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:34:52.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:34:52.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:34:52.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:34:52.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:34:52.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:34:52.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:34:52.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:34:52.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:34:52.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:34:52.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:34:52.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:34:52.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:34:52.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:34:52.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:34:52.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:34:52.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:34:52.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:34:52.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:34:52.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:34:52.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:34:52.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:34:52.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:34:52.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:34:52.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:34:52.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:34:52.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:34:52.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: return future.result(timeout) 2024-03-22T05:34:52.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:34:52.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.__get_result() 2024-03-22T05:34:52.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:34:52.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: raise self._exception 2024-03-22T05:34:52.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:34:52.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:34:52.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:34:52.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:34:52.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-iscsi-foo-smithi005-mcdwbp 2024-03-22T05:34:52.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: /usr/bin/podman: stderr Error: no such container ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-iscsi-foo-smithi005-mcdwbp 2024-03-22T05:34:52.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-iscsi.foo.smithi005.mcdwbp 2024-03-22T05:34:52.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: /usr/bin/podman: stderr Error: no such container ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-iscsi.foo.smithi005.mcdwbp 2024-03-22T05:34:52.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:34:52.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:34:52.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:34:52.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:34:52.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:34:52.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:34:52.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:34:52.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:34:52.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:34:52.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:34:52.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:34:52.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:34:52.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:34:52.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:34:52.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:34:52.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:34:52.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:34:52.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:34:52.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:34:52.799 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:34:52.799 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:34:54.596 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:54 smithi186 ceph-mon[25720]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 210 B/s rd, 0 op/s 2024-03-22T05:34:54.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:54 smithi005 ceph-mon[24793]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 210 B/s rd, 0 op/s 2024-03-22T05:34:54.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:54 smithi005 ceph-mon[29523]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 210 B/s rd, 0 op/s 2024-03-22T05:34:56.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:56 smithi005 ceph-mon[24793]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 210 B/s rd, 0 op/s 2024-03-22T05:34:56.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:56 smithi005 ceph-mon[29523]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 210 B/s rd, 0 op/s 2024-03-22T05:34:56.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:56 smithi186 ceph-mon[25720]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 210 B/s rd, 0 op/s 2024-03-22T05:34:57.314 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:34:56 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:34:56] "GET /metrics HTTP/1.1" 200 37967 "" "Prometheus/2.33.4" 2024-03-22T05:34:57.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:57 smithi005 ceph-mon[24793]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:34:57.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:57 smithi005 ceph-mon[29523]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:34:57.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:57 smithi186 ceph-mon[25720]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:34:58.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:34:58 smithi005 ceph-mon[24793]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2024-03-22T05:34:58.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:34:58 smithi005 ceph-mon[29523]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2024-03-22T05:34:58.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:34:58 smithi186 ceph-mon[25720]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2024-03-22T05:35:00.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:35:00 smithi005 ceph-mon[24793]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-03-22T05:35:00.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:35:00 smithi005 ceph-mon[29523]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-03-22T05:35:00.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:35:00 smithi186 ceph-mon[25720]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-03-22T05:35:02.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:35:02 smithi005 ceph-mon[29523]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-03-22T05:35:02.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:35:02 smithi005 ceph-mon[29523]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:35:02.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:35:02 smithi005 ceph-mon[24793]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-03-22T05:35:02.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:35:02 smithi005 ceph-mon[24793]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:35:02.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:35:02 smithi186 ceph-mon[25720]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-03-22T05:35:02.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:35:02 smithi186 ceph-mon[25720]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:35:04.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:35:04 smithi005 ceph-mon[29523]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-03-22T05:35:04.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:35:04 smithi005 ceph-mon[24793]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-03-22T05:35:04.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:35:04 smithi186 ceph-mon[25720]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-03-22T05:35:06.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:35:06 smithi005 ceph-mon[24793]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:35:06.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:35:06 smithi005 ceph-mon[29523]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:35:06.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:35:06 smithi186 ceph-mon[25720]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:35:07.452 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:35:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:35:06] "GET /metrics HTTP/1.1" 200 37967 "" "Prometheus/2.33.4" 2024-03-22T05:35:08.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:35:08 smithi005 ceph-mon[24793]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:35:08.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:35:08 smithi005 ceph-mon[29523]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:35:08.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:35:08 smithi186 ceph-mon[25720]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:35:10.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:35:10 smithi005 ceph-mon[24793]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:35:10.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:35:10 smithi005 ceph-mon[29523]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:35:10.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:35:10 smithi186 ceph-mon[25720]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:35:12.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:35:12 smithi005 ceph-mon[29523]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:35:12.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:35:12 smithi005 ceph-mon[29523]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:35:12.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:35:12 smithi005 ceph-mon[29523]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:35:12.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:35:12 smithi005 ceph-mon[24793]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:35:12.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:35:12 smithi005 ceph-mon[24793]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:35:12.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:35:12 smithi005 ceph-mon[24793]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:35:12.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:35:12 smithi186 ceph-mon[25720]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:35:12.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:35:12 smithi186 ceph-mon[25720]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:35:12.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:35:12 smithi186 ceph-mon[25720]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:35:14.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:35:14 smithi005 ceph-mon[24793]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:35:14.705 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:35:14 smithi005 ceph-mon[29523]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:35:14.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:35:14 smithi186 ceph-mon[25720]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:35:16.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:35:16 smithi005 ceph-mon[24793]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:35:16.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:35:16 smithi005 ceph-mon[29523]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:35:16.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:35:16 smithi186 ceph-mon[25720]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:35:17.452 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:35:16 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:35:16] "GET /metrics HTTP/1.1" 200 37968 "" "Prometheus/2.33.4" 2024-03-22T05:35:18.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:35:18 smithi005 ceph-mon[29523]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:35:18.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:35:18 smithi005 ceph-mon[24793]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:35:18.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:35:18 smithi186 ceph-mon[25720]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:35:20.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:35:20 smithi005 ceph-mon[29523]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:35:20.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:35:20 smithi005 ceph-mon[24793]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:35:20.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:35:20 smithi186 ceph-mon[25720]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:35:22.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:35:22 smithi005 ceph-mon[24793]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:35:22.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:35:22 smithi005 ceph-mon[24793]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:35:22.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:35:22 smithi005 ceph-mon[29523]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:35:22.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:35:22 smithi005 ceph-mon[29523]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:35:22.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:35:22 smithi186 ceph-mon[25720]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:35:22.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:35:22 smithi186 ceph-mon[25720]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:35:24.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:35:24 smithi005 ceph-mon[24793]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:35:24.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:35:24 smithi005 ceph-mon[29523]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:35:24.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:35:24 smithi186 ceph-mon[25720]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:35:26.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:35:26 smithi005 ceph-mon[24793]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:35:26.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:35:26 smithi005 ceph-mon[29523]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:35:26.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:35:26 smithi186 ceph-mon[25720]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:35:27.357 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:35:26 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:35:26] "GET /metrics HTTP/1.1" 200 37969 "" "Prometheus/2.33.4" 2024-03-22T05:35:27.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:35:27 smithi005 ceph-mon[24793]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:35:27.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:35:27 smithi005 ceph-mon[29523]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:35:27.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:35:27 smithi186 ceph-mon[25720]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:35:28.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:35:28 smithi005 ceph-mon[24793]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:35:28.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:35:28 smithi005 ceph-mon[29523]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:35:28.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:35:28 smithi186 ceph-mon[25720]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:35:30.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:35:30 smithi005 ceph-mon[29523]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:35:30.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:35:30 smithi005 ceph-mon[24793]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:35:30.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:35:30 smithi186 ceph-mon[25720]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:35:32.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:35:32 smithi005 ceph-mon[29523]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:35:32.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:35:32 smithi005 ceph-mon[29523]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:35:32.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:35:32 smithi005 ceph-mon[24793]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:35:32.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:35:32 smithi005 ceph-mon[24793]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:35:32.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:35:32 smithi186 ceph-mon[25720]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:35:32.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:35:32 smithi186 ceph-mon[25720]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:35:34.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:35:34 smithi005 ceph-mon[24793]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:35:34.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:35:34 smithi005 ceph-mon[29523]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:35:34.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:35:34 smithi186 ceph-mon[25720]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:35:36.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:35:36 smithi005 ceph-mon[24793]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:35:36.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:35:36 smithi005 ceph-mon[29523]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:35:36.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:35:36 smithi186 ceph-mon[25720]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:35:37.452 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:35:36 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:35:36] "GET /metrics HTTP/1.1" 200 37969 "" "Prometheus/2.33.4" 2024-03-22T05:35:38.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:35:38 smithi005 ceph-mon[24793]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:35:38.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:35:38 smithi005 ceph-mon[29523]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:35:38.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:35:38 smithi186 ceph-mon[25720]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:35:40.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:35:40 smithi005 ceph-mon[29523]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:35:40.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:35:40 smithi005 ceph-mon[24793]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:35:40.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:35:40 smithi186 ceph-mon[25720]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:35:42.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:35:42 smithi005 ceph-mon[29523]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:35:42.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:35:42 smithi005 ceph-mon[29523]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:35:42.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:35:42 smithi005 ceph-mon[29523]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:35:42.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:35:42 smithi005 ceph-mon[24793]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:35:42.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:35:42 smithi005 ceph-mon[24793]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:35:42.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:35:42 smithi005 ceph-mon[24793]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:35:42.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:35:42 smithi186 ceph-mon[25720]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:35:42.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:35:42 smithi186 ceph-mon[25720]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:35:42.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:35:42 smithi186 ceph-mon[25720]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:35:44.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:35:44 smithi005 ceph-mon[24793]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:35:44.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:35:44 smithi005 ceph-mon[29523]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:35:44.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:35:44 smithi186 ceph-mon[25720]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:35:46.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:35:46 smithi005 ceph-mon[29523]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:35:46.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:35:46 smithi005 ceph-mon[24793]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:35:46.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:35:46 smithi186 ceph-mon[25720]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:35:47.452 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:35:46 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:35:46] "GET /metrics HTTP/1.1" 200 37967 "" "Prometheus/2.33.4" 2024-03-22T05:35:48.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:35:48 smithi005 ceph-mon[24793]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:35:48.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:35:48 smithi005 ceph-mon[29523]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:35:48.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:35:48 smithi186 ceph-mon[25720]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:35:50.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:35:50 smithi005 ceph-mon[29523]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:35:50.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:35:50 smithi005 ceph-mon[24793]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:35:50.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:35:50 smithi186 ceph-mon[25720]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:35:52.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:35:52 smithi005 ceph-mon[29523]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:35:52.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:35:52 smithi005 ceph-mon[29523]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:35:52.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:35:52 smithi005 ceph-mon[29523]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:35:52.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:35:52 smithi005 ceph-mon[29523]: from='mgr.25111 ' entity='mgr.y' 2024-03-22T05:35:52.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:35:52 smithi005 ceph-mon[29523]: from='mgr.25111 ' entity='mgr.y' 2024-03-22T05:35:52.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:35:52 smithi005 ceph-mon[29523]: from='mgr.25111 ' entity='mgr.y' 2024-03-22T05:35:52.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:35:52 smithi005 ceph-mon[29523]: from='mgr.25111 ' entity='mgr.y' 2024-03-22T05:35:52.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:35:52 smithi005 ceph-mon[24793]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:35:52.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:35:52 smithi005 ceph-mon[24793]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:35:52.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:35:52 smithi005 ceph-mon[24793]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:35:52.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:35:52 smithi005 ceph-mon[24793]: from='mgr.25111 ' entity='mgr.y' 2024-03-22T05:35:52.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:35:52 smithi005 ceph-mon[24793]: from='mgr.25111 ' entity='mgr.y' 2024-03-22T05:35:52.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:35:52 smithi005 ceph-mon[24793]: from='mgr.25111 ' entity='mgr.y' 2024-03-22T05:35:52.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:35:52 smithi005 ceph-mon[24793]: from='mgr.25111 ' entity='mgr.y' 2024-03-22T05:35:52.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:35:52 smithi186 ceph-mon[25720]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:35:52.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:35:52 smithi186 ceph-mon[25720]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:35:52.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:35:52 smithi186 ceph-mon[25720]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:35:52.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:35:52 smithi186 ceph-mon[25720]: from='mgr.25111 ' entity='mgr.y' 2024-03-22T05:35:52.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:35:52 smithi186 ceph-mon[25720]: from='mgr.25111 ' entity='mgr.y' 2024-03-22T05:35:52.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:35:52 smithi186 ceph-mon[25720]: from='mgr.25111 ' entity='mgr.y' 2024-03-22T05:35:52.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:35:52 smithi186 ceph-mon[25720]: from='mgr.25111 ' entity='mgr.y' 2024-03-22T05:35:53.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:35:53 smithi005 ceph-mon[29523]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:35:53.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:35:53 smithi005 ceph-mon[29523]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:35:53.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:35:53 smithi005 ceph-mon[29523]: from='mgr.25111 ' entity='mgr.y' 2024-03-22T05:35:53.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:35:53 smithi005 ceph-mon[29523]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:35:53.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:35:53 smithi005 ceph-mon[29523]: from='mgr.25111 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:35:53.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:35:53 smithi005 ceph-mon[29523]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:35:53.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:35:53 smithi005 ceph-mon[24793]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:35:53.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:35:53 smithi005 ceph-mon[24793]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:35:53.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:35:53 smithi005 ceph-mon[24793]: from='mgr.25111 ' entity='mgr.y' 2024-03-22T05:35:53.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:35:53 smithi005 ceph-mon[24793]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:35:53.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:35:53 smithi005 ceph-mon[24793]: from='mgr.25111 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:35:53.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:35:53 smithi005 ceph-mon[24793]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:35:53.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:35:53 smithi186 ceph-mon[25720]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:35:53.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:35:53 smithi186 ceph-mon[25720]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:35:53.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:35:53 smithi186 ceph-mon[25720]: from='mgr.25111 ' entity='mgr.y' 2024-03-22T05:35:53.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:35:53 smithi186 ceph-mon[25720]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:35:53.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:35:53 smithi186 ceph-mon[25720]: from='mgr.25111 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:35:53.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:35:53 smithi186 ceph-mon[25720]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:35:54.681 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:35:54 smithi005 ceph-mon[24793]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-22T05:35:54.681 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:35:54 smithi005 ceph-mon[24793]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-22T05:35:54.681 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:35:54 smithi005 ceph-mon[24793]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:35:54.681 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:35:54 smithi005 ceph-mon[24793]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:35:54.682 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:35:54 smithi005 ceph-mon[29523]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-22T05:35:54.682 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:35:54 smithi005 ceph-mon[29523]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-22T05:35:54.682 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:35:54 smithi005 ceph-mon[29523]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:35:54.682 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:35:54 smithi005 ceph-mon[29523]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:35:54.695 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:35:54 smithi186 ceph-mon[25720]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-22T05:35:54.695 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:35:54 smithi186 ceph-mon[25720]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-22T05:35:54.695 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:35:54 smithi186 ceph-mon[25720]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:35:54.695 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:35:54 smithi186 ceph-mon[25720]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:35:54.952 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:35:54 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:35:54.820+0000 7f8428d6c700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:35:54.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:35:54 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:35:54.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:35:54 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:35:54.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:35:54 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:35:54.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:35:54 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:35:54.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:35:54 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:35:54.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:35:54 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:35:54.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:35:54 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:35:54.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:35:54 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:35:54.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:35:54 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:35:54.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:35:54 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:35:54.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:35:54 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:35:54.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:35:54 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:35:54.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:35:54 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:35:54.954 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:35:54 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:35:54.954 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:35:54 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:35:54.954 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:35:54 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:35:54.954 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:35:54 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:35:54.954 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:35:54 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:35:54.954 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:35:54 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:35:54.954 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:35:54 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:35:54.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:35:54 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:35:54.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:35:54 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:35:54.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:35:54 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:35:54.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:35:54 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:35:54.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:35:54 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:35:54.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:35:54 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:35:54.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:35:54 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:35:54.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:35:54 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:35:54.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:35:54 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return future.result(timeout) 2024-03-22T05:35:54.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:35:54 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:35:54.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:35:54 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.__get_result() 2024-03-22T05:35:54.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:35:54 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:35:54.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:35:54 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: raise self._exception 2024-03-22T05:35:54.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:35:54 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:35:54.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:35:54 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: config_blobs=daemon_spec.final_config, 2024-03-22T05:35:54.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:35:54 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:35:54.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:35:54 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:35:54.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:35:54 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:35:54.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:35:54 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:35:54.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:35:54 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:35:54.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:35:54 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:35:54.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:35:54 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:35:54.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:35:54 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:35:54.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:35:54 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:35:54.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:35:54 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:35:54.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:35:54 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:35:54.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:35:54 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:35:54.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:35:54 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:35:54.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:35:54 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:35:54.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:35:54 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:35:54.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:35:54 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:35:54.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:35:54 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:35:54.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:35:54 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:35:54.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:35:54 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:35:54.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:35:54 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:35:54.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:35:54 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:35:54.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:35:54 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:35:54.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:35:54 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:35:56.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:35:56 smithi005 ceph-mon[24793]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:35:56.702 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:35:56.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:35:56.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:35:56.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:35:56.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:35:56.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:35:56.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:35:56.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:35:56.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:35:56.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:35:56.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:35:56.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:35:56.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:35:56.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:35:56.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:35:56.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:35:56.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:35:56.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:35:56.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:35:56.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:35:56.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:35:56.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:35:56.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:35:56.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:35:56.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:35:56.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:35:56.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:35:56.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:35:56.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: return future.result(timeout) 2024-03-22T05:35:56.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:35:56.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.__get_result() 2024-03-22T05:35:56.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:35:56.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: raise self._exception 2024-03-22T05:35:56.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:35:56.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:35:56.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:35:56.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:35:56.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:35:56.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:35:56.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:35:56.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:35:56.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:35:56.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:35:56.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:35:56.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:35:56.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:35:56.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:35:56.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:35:56.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:35:56.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:35:56.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:35:56.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:35:56.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:35:56.706 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:35:56 smithi005 ceph-mon[29523]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:35:56.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:35:56.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:35:56.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:35:56.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:35:56.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:35:56.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:35:56.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:35:56.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:35:56.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:35:56.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:35:56.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:35:56.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:35:56.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:35:56.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:35:56.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:35:56.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:35:56.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:35:56.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:35:56.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:35:56.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:35:56.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:35:56.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:35:56.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:35:56.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:35:56.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:35:56.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:35:56.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:35:56.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:35:56.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: return future.result(timeout) 2024-03-22T05:35:56.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:35:56.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.__get_result() 2024-03-22T05:35:56.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:35:56.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: raise self._exception 2024-03-22T05:35:56.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:35:56.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:35:56.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:35:56.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:35:56.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:35:56.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:35:56.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:35:56.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:35:56.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:35:56.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:35:56.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:35:56.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:35:56.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:35:56.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:35:56.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:35:56.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:35:56.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:35:56.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:35:56.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:35:56.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:35:56.710 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:35:56.711 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:35:56.711 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:35:56.711 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:35:56.711 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:35:56.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:35:56 smithi005 ceph-mon[24793]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-22T05:35:56.711 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:35:56.711 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:35:56.711 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:35:56.711 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:35:56.711 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:35:56.711 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:35:56 smithi005 ceph-mon[29523]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-22T05:35:56.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:35:56 smithi186 ceph-mon[25720]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:35:56.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:35:56.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:35:56.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:35:56.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:35:56.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:35:56.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:35:56.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:35:56.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:35:56.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:35:56.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:35:56.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:35:56.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:35:56.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:35:56.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:35:56.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:35:56.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:35:56.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:35:56.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:35:56.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:35:56.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:35:56.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:35:56.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:35:56.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:35:56.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:35:56.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:35:56.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:35:56.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:35:56.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:35:56.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: return future.result(timeout) 2024-03-22T05:35:56.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:35:56.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.__get_result() 2024-03-22T05:35:56.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:35:56.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: raise self._exception 2024-03-22T05:35:56.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:35:56.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:35:56.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:35:56.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:35:56.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:35:56.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:35:56.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:35:56.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:35:56.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:35:56.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:35:56.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:35:56.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:35:56.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:35:56.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:35:56.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:35:56.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:35:56.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:35:56.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:35:56.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:35:56.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:35:56.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:35:56.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:35:56.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:35:56.799 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:35:56.799 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:35:56.799 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:35:56 smithi186 ceph-mon[25720]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-22T05:35:57.407 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:35:56 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:35:56] "GET /metrics HTTP/1.1" 200 37962 "" "Prometheus/2.33.4" 2024-03-22T05:35:57.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:35:57 smithi005 ceph-mon[29523]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:35:57.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:35:57 smithi005 ceph-mon[24793]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:35:57.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:35:57 smithi186 ceph-mon[25720]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:35:58.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:35:58 smithi005 ceph-mon[24793]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-22T05:35:58.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:35:58 smithi005 ceph-mon[29523]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-22T05:35:58.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:35:58 smithi186 ceph-mon[25720]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-22T05:36:00.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:36:00 smithi005 ceph-mon[24793]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-22T05:36:00.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:36:00 smithi005 ceph-mon[29523]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-22T05:36:00.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:36:00 smithi186 ceph-mon[25720]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-22T05:36:02.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:36:02 smithi005 ceph-mon[24793]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-22T05:36:02.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:36:02 smithi005 ceph-mon[24793]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:36:02.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:36:02 smithi005 ceph-mon[29523]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-22T05:36:02.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:36:02 smithi005 ceph-mon[29523]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:36:02.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:36:02 smithi186 ceph-mon[25720]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-22T05:36:02.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:36:02 smithi186 ceph-mon[25720]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:36:04.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:36:04 smithi005 ceph-mon[24793]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:36:04.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:36:04 smithi005 ceph-mon[29523]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:36:04.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:36:04 smithi186 ceph-mon[25720]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:36:06.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:36:06 smithi005 ceph-mon[24793]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:36:06.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:36:06 smithi005 ceph-mon[29523]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:36:06.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:36:06 smithi186 ceph-mon[25720]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:36:07.452 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:36:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:36:06] "GET /metrics HTTP/1.1" 200 37962 "" "Prometheus/2.33.4" 2024-03-22T05:36:08.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:36:08 smithi005 ceph-mon[24793]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:36:08.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:36:08 smithi005 ceph-mon[29523]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:36:08.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:36:08 smithi186 ceph-mon[25720]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:36:10.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:36:10 smithi005 ceph-mon[24793]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:36:10.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:36:10 smithi005 ceph-mon[29523]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:36:10.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:36:10 smithi186 ceph-mon[25720]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:36:12.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:36:12 smithi005 ceph-mon[24793]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:36:12.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:36:12 smithi005 ceph-mon[24793]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:36:12.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:36:12 smithi005 ceph-mon[24793]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:36:12.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:36:12 smithi005 ceph-mon[29523]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:36:12.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:36:12 smithi005 ceph-mon[29523]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:36:12.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:36:12 smithi005 ceph-mon[29523]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:36:12.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:36:12 smithi186 ceph-mon[25720]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:36:12.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:36:12 smithi186 ceph-mon[25720]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:36:12.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:36:12 smithi186 ceph-mon[25720]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:36:14.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:36:14 smithi005 ceph-mon[24793]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:36:14.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:36:14 smithi005 ceph-mon[29523]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:36:14.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:36:14 smithi186 ceph-mon[25720]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:36:16.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:36:16 smithi005 ceph-mon[29523]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:36:16.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:36:16 smithi005 ceph-mon[24793]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:36:16.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:36:16 smithi186 ceph-mon[25720]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:36:17.452 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:36:16 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:36:16] "GET /metrics HTTP/1.1" 200 37962 "" "Prometheus/2.33.4" 2024-03-22T05:36:18.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:36:18 smithi005 ceph-mon[24793]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:36:18.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:36:18 smithi005 ceph-mon[29523]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:36:18.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:36:18 smithi186 ceph-mon[25720]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:36:20.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:36:20 smithi005 ceph-mon[24793]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:36:20.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:36:20 smithi005 ceph-mon[29523]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:36:20.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:36:20 smithi186 ceph-mon[25720]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:36:22.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:36:22 smithi005 ceph-mon[24793]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:36:22.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:36:22 smithi005 ceph-mon[24793]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:36:22.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:36:22 smithi005 ceph-mon[29523]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:36:22.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:36:22 smithi005 ceph-mon[29523]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:36:22.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:36:22 smithi186 ceph-mon[25720]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:36:22.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:36:22 smithi186 ceph-mon[25720]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:36:24.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:36:24 smithi005 ceph-mon[24793]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:36:24.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:36:24 smithi005 ceph-mon[29523]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:36:24.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:36:24 smithi186 ceph-mon[25720]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:36:26.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:36:26 smithi005 ceph-mon[24793]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:36:26.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:36:26 smithi005 ceph-mon[29523]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:36:26.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:36:26 smithi186 ceph-mon[25720]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:36:27.452 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:36:26 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:36:26] "GET /metrics HTTP/1.1" 200 37958 "" "Prometheus/2.33.4" 2024-03-22T05:36:27.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:36:27 smithi186 ceph-mon[25720]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:36:27.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:36:27 smithi005 ceph-mon[24793]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:36:27.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:36:27 smithi005 ceph-mon[29523]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:36:28.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:36:28 smithi186 ceph-mon[25720]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:36:28.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:36:28 smithi005 ceph-mon[29523]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:36:28.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:36:28 smithi005 ceph-mon[24793]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:36:30.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:36:30 smithi186 ceph-mon[25720]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:36:30.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:36:30 smithi005 ceph-mon[24793]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:36:30.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:36:30 smithi005 ceph-mon[29523]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:36:32.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:36:32 smithi186 ceph-mon[25720]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:36:32.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:36:32 smithi186 ceph-mon[25720]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:36:32.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:36:32 smithi005 ceph-mon[24793]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:36:32.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:36:32 smithi005 ceph-mon[24793]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:36:32.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:36:32 smithi005 ceph-mon[29523]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:36:32.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:36:32 smithi005 ceph-mon[29523]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:36:34.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:36:34 smithi186 ceph-mon[25720]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:36:34.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:36:34 smithi005 ceph-mon[24793]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:36:34.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:36:34 smithi005 ceph-mon[29523]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:36:36.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:36:36 smithi186 ceph-mon[25720]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:36:36.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:36:36 smithi005 ceph-mon[24793]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:36:36.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:36:36 smithi005 ceph-mon[29523]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:36:37.452 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:36:36 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:36:36] "GET /metrics HTTP/1.1" 200 37958 "" "Prometheus/2.33.4" 2024-03-22T05:36:38.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:36:38 smithi186 ceph-mon[25720]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:36:38.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:36:38 smithi005 ceph-mon[24793]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:36:38.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:36:38 smithi005 ceph-mon[29523]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:36:40.754 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:36:40 smithi005 ceph-mon[29523]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:36:40.755 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:36:40 smithi005 ceph-mon[24793]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:36:40.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:36:40 smithi186 ceph-mon[25720]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:36:42.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:36:42 smithi186 ceph-mon[25720]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:36:42.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:36:42 smithi186 ceph-mon[25720]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:36:42.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:36:42 smithi186 ceph-mon[25720]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:36:42.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:36:42 smithi005 ceph-mon[24793]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:36:42.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:36:42 smithi005 ceph-mon[24793]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:36:42.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:36:42 smithi005 ceph-mon[24793]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:36:42.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:36:42 smithi005 ceph-mon[29523]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:36:42.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:36:42 smithi005 ceph-mon[29523]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:36:42.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:36:42 smithi005 ceph-mon[29523]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:36:44.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:36:44 smithi186 ceph-mon[25720]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:36:44.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:36:44 smithi005 ceph-mon[24793]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:36:44.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:36:44 smithi005 ceph-mon[29523]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:36:46.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:36:46 smithi186 ceph-mon[25720]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:36:46.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:36:46 smithi005 ceph-mon[29523]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:36:46.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:36:46 smithi005 ceph-mon[24793]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:36:47.452 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:36:46 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:36:46] "GET /metrics HTTP/1.1" 200 37977 "" "Prometheus/2.33.4" 2024-03-22T05:36:48.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:36:48 smithi186 ceph-mon[25720]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:36:48.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:36:48 smithi005 ceph-mon[24793]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:36:48.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:36:48 smithi005 ceph-mon[29523]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:36:50.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:36:50 smithi186 ceph-mon[25720]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:36:50.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:36:50 smithi005 ceph-mon[24793]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:36:50.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:36:50 smithi005 ceph-mon[29523]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:36:52.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:36:52 smithi186 ceph-mon[25720]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:36:52.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:36:52 smithi186 ceph-mon[25720]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:36:52.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:36:52 smithi005 ceph-mon[24793]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:36:52.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:36:52 smithi005 ceph-mon[24793]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:36:52.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:36:52 smithi005 ceph-mon[29523]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:36:52.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:36:52 smithi005 ceph-mon[29523]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:36:54.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:36:54 smithi186 ceph-mon[25720]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:36:54.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:36:54 smithi005 ceph-mon[24793]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:36:54.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:36:54 smithi005 ceph-mon[29523]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:36:55.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:36:55 smithi186 ceph-mon[25720]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:36:55.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:36:55 smithi005 ceph-mon[24793]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:36:55.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:36:55 smithi005 ceph-mon[29523]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:36:56.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:36:56 smithi005 ceph-mon[29523]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:36:56.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:36:56 smithi005 ceph-mon[29523]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:36:56.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:36:56 smithi005 ceph-mon[29523]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:36:56.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:36:56 smithi005 ceph-mon[29523]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 961 B/s rd, 0 op/s 2024-03-22T05:36:56.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:36:56 smithi005 ceph-mon[29523]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:36:56.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:36:56 smithi005 ceph-mon[29523]: from='mgr.25111 ' entity='mgr.y' 2024-03-22T05:36:56.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:36:56 smithi005 ceph-mon[29523]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:36:56.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:36:56 smithi005 ceph-mon[29523]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:36:56.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:36:56 smithi005 ceph-mon[29523]: from='mgr.25111 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:36:56.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:36:56 smithi005 ceph-mon[29523]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:36:56.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:36:56 smithi005 ceph-mon[29523]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:36:56.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:36:56 smithi005 ceph-mon[24793]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:36:56.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:36:56 smithi005 ceph-mon[24793]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:36:56.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:36:56 smithi005 ceph-mon[24793]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:36:56.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:36:56 smithi005 ceph-mon[24793]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 961 B/s rd, 0 op/s 2024-03-22T05:36:56.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:36:56 smithi005 ceph-mon[24793]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:36:56.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:36:56 smithi005 ceph-mon[24793]: from='mgr.25111 ' entity='mgr.y' 2024-03-22T05:36:56.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:36:56 smithi005 ceph-mon[24793]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:36:56.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:36:56 smithi005 ceph-mon[24793]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:36:56.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:36:56 smithi005 ceph-mon[24793]: from='mgr.25111 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:36:56.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:36:56 smithi005 ceph-mon[24793]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:36:56.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:36:56 smithi005 ceph-mon[24793]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:36:56.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:36:56 smithi186 ceph-mon[25720]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:36:56.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:36:56 smithi186 ceph-mon[25720]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:36:56.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:36:56 smithi186 ceph-mon[25720]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:36:56.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:36:56 smithi186 ceph-mon[25720]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 961 B/s rd, 0 op/s 2024-03-22T05:36:56.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:36:56 smithi186 ceph-mon[25720]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:36:56.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:36:56 smithi186 ceph-mon[25720]: from='mgr.25111 ' entity='mgr.y' 2024-03-22T05:36:56.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:36:56 smithi186 ceph-mon[25720]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:36:56.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:36:56 smithi186 ceph-mon[25720]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:36:56.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:36:56 smithi186 ceph-mon[25720]: from='mgr.25111 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:36:56.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:36:56 smithi186 ceph-mon[25720]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:36:56.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:36:56 smithi186 ceph-mon[25720]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:36:57.198 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:36:56 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:36:56] "GET /metrics HTTP/1.1" 200 37974 "" "Prometheus/2.33.4" 2024-03-22T05:36:57.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:36:57 smithi186 ceph-mon[25720]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:36:57.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:36:57 smithi005 ceph-mon[24793]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:36:57.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:36:57 smithi005 ceph-mon[29523]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:36:57.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:36:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:36:57.589+0000 7f8428d6c700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:36:57.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:36:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:36:57.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:36:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:36:57.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:36:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:36:57.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:36:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:36:57.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:36:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:36:57.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:36:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:36:57.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:36:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:36:57.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:36:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:36:57.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:36:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:36:57.954 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:36:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:36:57.954 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:36:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:36:57.954 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:36:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:36:57.954 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:36:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:36:57.954 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:36:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:36:57.954 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:36:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:36:57.954 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:36:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:36:57.954 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:36:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:36:57.954 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:36:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:36:57.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:36:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:36:57.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:36:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:36:57.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:36:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:36:57.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:36:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:36:57.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:36:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:36:57.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:36:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:36:57.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:36:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:36:57.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:36:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:36:57.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:36:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:36:57.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:36:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:36:57.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:36:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return future.result(timeout) 2024-03-22T05:36:57.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:36:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:36:57.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:36:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.__get_result() 2024-03-22T05:36:57.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:36:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:36:57.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:36:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: raise self._exception 2024-03-22T05:36:57.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:36:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:36:57.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:36:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: config_blobs=daemon_spec.final_config, 2024-03-22T05:36:57.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:36:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:36:57.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:36:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:36:57.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:36:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:36:57.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:36:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:36:57.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:36:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:36:57.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:36:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:36:57.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:36:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:36:57.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:36:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:36:57.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:36:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:36:57.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:36:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:36:57.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:36:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:36:57.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:36:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:36:57.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:36:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:36:57.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:36:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:36:57.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:36:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:36:57.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:36:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:36:57.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:36:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:36:57.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:36:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:36:57.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:36:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:36:57.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:36:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:36:57.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:36:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:36:57.958 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:36:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:36:57.958 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:36:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:36:58.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:36:58 smithi186 ceph-mon[25720]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 591 B/s rd, 0 op/s 2024-03-22T05:36:58.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:36:58 smithi186 ceph-mon[25720]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:36:58.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:36:58.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:36:58.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:36:58.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:36:58.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:36:58.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:36:58.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:36:58.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:36:58.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:36:58.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:36:58.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:36:58.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:36:58.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:36:58.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:36:58.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:36:58.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:36:58.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:36:58.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:36:58.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:36:58.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:36:58.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:36:58.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:36:58.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:36:58.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:36:58.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:36:58.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:36:58.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:36:58.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:36:58.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: return future.result(timeout) 2024-03-22T05:36:58.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:36:58.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.__get_result() 2024-03-22T05:36:58.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:36:58.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: raise self._exception 2024-03-22T05:36:58.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:36:58.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:36:58.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:36:58.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:36:58.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:36:58.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:36:58.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:36:58.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:36:58.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:36:58.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:36:58.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:36:58.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:36:58.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:36:58.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:36:58.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:36:58.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:36:58.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:36:58.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:36:58.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:36:58.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:36:58.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:36:58.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:36:58.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:36:58.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:36:58.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:36:58.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:36:58 smithi005 ceph-mon[24793]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 591 B/s rd, 0 op/s 2024-03-22T05:36:58.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:36:58 smithi005 ceph-mon[24793]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:36:58.952 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:36:58.952 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:36:58.952 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:36:58.952 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:36:58.952 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:36:58.952 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:36:58.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:36:58.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:36:58.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:36:58.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:36:58.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:36:58.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:36:58.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:36:58.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:36:58.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:36:58.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:36:58.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:36:58.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:36:58.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:36:58.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:36:58.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:36:58.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:36:58.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:36:58.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:36:58.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:36:58.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:36:58.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:36:58.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:36:58.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: return future.result(timeout) 2024-03-22T05:36:58.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:36:58.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.__get_result() 2024-03-22T05:36:58.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:36:58.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: raise self._exception 2024-03-22T05:36:58.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:36:58.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:36:58.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:36:58.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:36:58.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:36:58.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:36:58.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:36:58.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:36:58.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:36:58.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:36:58.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:36:58.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:36:58.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:36:58.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:36:58.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:36:58.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:36:58.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:36:58.956 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:36:58.956 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:36:58.956 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:36:58.956 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:36:58.956 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:36:58.956 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:36:58.956 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:36:58.956 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:36:58.956 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:36:58 smithi005 ceph-mon[29523]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 591 B/s rd, 0 op/s 2024-03-22T05:36:58.956 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:36:58 smithi005 ceph-mon[29523]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:36:58.956 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:36:58.956 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:36:58.956 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:36:58.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:36:58.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:36:58.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:36:58.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:36:58.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:36:58.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:36:58.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:36:58.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:36:58.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:36:58.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:36:58.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:36:58.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:36:58.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:36:58.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:36:58.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:36:58.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:36:58.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:36:58.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:36:58.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:36:58.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:36:58.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:36:58.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:36:58.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:36:58.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:36:58.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:36:58.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: return future.result(timeout) 2024-03-22T05:36:58.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:36:58.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.__get_result() 2024-03-22T05:36:58.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:36:58.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: raise self._exception 2024-03-22T05:36:58.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:36:58.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:36:58.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:36:58.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:36:58.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:36:58.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:36:58.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:36:58.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:36:58.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:36:58.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:36:58.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:36:58.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:36:58.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:36:58.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:36:58.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:36:58.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:36:58.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:36:58.960 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:36:58.960 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:36:58.960 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:36:58.960 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:36:58.960 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:36:58.960 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:36:58.960 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:36:58.960 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:37:00.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:00 smithi186 ceph-mon[25720]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:37:00.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:00 smithi005 ceph-mon[24793]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:37:00.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:00 smithi005 ceph-mon[29523]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:37:02.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:02 smithi186 ceph-mon[25720]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:37:02.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:02 smithi186 ceph-mon[25720]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 591 B/s rd, 0 op/s 2024-03-22T05:37:02.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:02 smithi005 ceph-mon[24793]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:37:02.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:02 smithi005 ceph-mon[24793]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 591 B/s rd, 0 op/s 2024-03-22T05:37:02.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:02 smithi005 ceph-mon[29523]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:37:02.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:02 smithi005 ceph-mon[29523]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 591 B/s rd, 0 op/s 2024-03-22T05:37:04.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:04 smithi005 ceph-mon[29523]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:37:04.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:04 smithi005 ceph-mon[24793]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:37:04.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:04 smithi186 ceph-mon[25720]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:37:06.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:06 smithi005 ceph-mon[29523]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:37:06.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:06 smithi005 ceph-mon[24793]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:37:07.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:06 smithi186 ceph-mon[25720]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:37:07.452 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:37:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:37:06] "GET /metrics HTTP/1.1" 200 37974 "" "Prometheus/2.33.4" 2024-03-22T05:37:08.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:08 smithi005 ceph-mon[29523]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:37:08.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:08 smithi005 ceph-mon[24793]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:37:09.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:08 smithi186 ceph-mon[25720]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:37:10.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:10 smithi005 ceph-mon[24793]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:37:10.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:10 smithi005 ceph-mon[29523]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:37:11.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:10 smithi186 ceph-mon[25720]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:37:12.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:12 smithi005 ceph-mon[24793]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:37:12.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:12 smithi005 ceph-mon[24793]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:37:12.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:12 smithi005 ceph-mon[24793]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:37:12.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:12 smithi005 ceph-mon[29523]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:37:12.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:12 smithi005 ceph-mon[29523]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:37:12.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:12 smithi005 ceph-mon[29523]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:37:13.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:12 smithi186 ceph-mon[25720]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:37:13.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:12 smithi186 ceph-mon[25720]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:37:13.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:12 smithi186 ceph-mon[25720]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:37:14.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:14 smithi005 ceph-mon[24793]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:37:14.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:14 smithi005 ceph-mon[29523]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:37:14.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:14 smithi186 ceph-mon[25720]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:37:16.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:16 smithi005 ceph-mon[24793]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:37:16.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:16 smithi005 ceph-mon[29523]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:37:17.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:16 smithi186 ceph-mon[25720]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:37:17.452 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:37:16 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:37:16] "GET /metrics HTTP/1.1" 200 37975 "" "Prometheus/2.33.4" 2024-03-22T05:37:18.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:18 smithi005 ceph-mon[24793]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:37:18.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:18 smithi005 ceph-mon[29523]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:37:19.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:18 smithi186 ceph-mon[25720]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:37:20.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:20 smithi005 ceph-mon[24793]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:37:20.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:20 smithi005 ceph-mon[29523]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:37:21.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:20 smithi186 ceph-mon[25720]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:37:22.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:22 smithi005 ceph-mon[29523]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:37:22.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:22 smithi005 ceph-mon[29523]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:37:22.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:22 smithi005 ceph-mon[24793]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:37:22.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:22 smithi005 ceph-mon[24793]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:37:23.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:22 smithi186 ceph-mon[25720]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:37:23.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:22 smithi186 ceph-mon[25720]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:37:24.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:24 smithi005 ceph-mon[24793]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:37:24.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:24 smithi005 ceph-mon[29523]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:37:24.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:24 smithi186 ceph-mon[25720]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:37:26.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:26 smithi005 ceph-mon[29523]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:37:26.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:26 smithi005 ceph-mon[24793]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:37:27.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:26 smithi186 ceph-mon[25720]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:37:27.452 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:37:26 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:37:26] "GET /metrics HTTP/1.1" 200 37973 "" "Prometheus/2.33.4" 2024-03-22T05:37:27.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:27 smithi005 ceph-mon[24793]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:37:27.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:27 smithi005 ceph-mon[29523]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:37:28.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:27 smithi186 ceph-mon[25720]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:37:28.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:28 smithi005 ceph-mon[24793]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:37:28.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:28 smithi005 ceph-mon[29523]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:37:29.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:28 smithi186 ceph-mon[25720]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:37:29.675 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph orch ps' 2024-03-22T05:37:30.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:30 smithi005 ceph-mon[24793]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:37:30.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:30 smithi005 ceph-mon[29523]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:37:31.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:30 smithi186 ceph-mon[25720]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:37:31.779 INFO:teuthology.orchestra.run.smithi005.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T05:37:31.779 INFO:teuthology.orchestra.run.smithi005.stdout:alertmanager.a smithi005 *:9093,9094 running (9m) 2m ago 16m 23.7M - 0.23.0 ba2b418f427c fd7dc5dbe3e5 2024-03-22T05:37:31.779 INFO:teuthology.orchestra.run.smithi005.stdout:grafana.a smithi186 *:3000 error 2m ago 16m - - 2024-03-22T05:37:31.779 INFO:teuthology.orchestra.run.smithi005.stdout:iscsi.foo.smithi005.mcdwbp smithi005 running (6m) 2m ago 16m 49.0M - 3.5 e1d6a67b021e 9d19a0c36be9 2024-03-22T05:37:31.779 INFO:teuthology.orchestra.run.smithi005.stdout:mgr.x smithi186 *:8443,9283 running (11m) 2m ago 21m 448M - 19.0.0-2299-gea01217f aa461e7c5e7f 66924e151e7b 2024-03-22T05:37:31.780 INFO:teuthology.orchestra.run.smithi005.stdout:mgr.y smithi005 *:8443,9283,8765 running (6m) 2m ago 23m 475M - 19.0.0-2299-gea01217f aa461e7c5e7f 62a813eebc0d 2024-03-22T05:37:31.780 INFO:teuthology.orchestra.run.smithi005.stdout:mon.a smithi005 running (23m) 2m ago 23m 61.9M 2048M 17.2.0 e1d6a67b021e 38e0bba4bb03 2024-03-22T05:37:31.780 INFO:teuthology.orchestra.run.smithi005.stdout:mon.b smithi186 running (21m) 2m ago 21m 45.4M 2048M 17.2.0 e1d6a67b021e 323f53f932f6 2024-03-22T05:37:31.780 INFO:teuthology.orchestra.run.smithi005.stdout:mon.c smithi005 running (21m) 2m ago 21m 44.7M 2048M 17.2.0 e1d6a67b021e 00ec24d38612 2024-03-22T05:37:31.780 INFO:teuthology.orchestra.run.smithi005.stdout:node-exporter.a smithi005 *:9100 running (9m) 2m ago 17m 19.8M - 1.3.1 1dbe0e931976 a9085d6042eb 2024-03-22T05:37:31.780 INFO:teuthology.orchestra.run.smithi005.stdout:node-exporter.b smithi186 *:9100 running (8m) 2m ago 17m 21.7M - 1.3.1 1dbe0e931976 74aa1181ff71 2024-03-22T05:37:31.780 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0 smithi005 running (21m) 2m ago 21m 52.3M 2993M 17.2.0 e1d6a67b021e ef5c7053dd37 2024-03-22T05:37:31.780 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1 smithi005 running (20m) 2m ago 20m 55.0M 2993M 17.2.0 e1d6a67b021e a1e7ee1017b6 2024-03-22T05:37:31.780 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2 smithi005 running (20m) 2m ago 20m 47.9M 2993M 17.2.0 e1d6a67b021e 462ea63ad0e5 2024-03-22T05:37:31.780 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3 smithi005 running (19m) 2m ago 19m 55.1M 2993M 17.2.0 e1d6a67b021e b25432a7239b 2024-03-22T05:37:31.780 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4 smithi186 running (19m) 2m ago 19m 53.6M 3248M 17.2.0 e1d6a67b021e cd963671f340 2024-03-22T05:37:31.780 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5 smithi186 running (19m) 2m ago 19m 50.7M 3248M 17.2.0 e1d6a67b021e f98c87f1a2cd 2024-03-22T05:37:31.780 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6 smithi186 running (18m) 2m ago 18m 49.9M 3248M 17.2.0 e1d6a67b021e 704ac49c68c4 2024-03-22T05:37:31.780 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7 smithi186 running (18m) 2m ago 18m 52.5M 3248M 17.2.0 e1d6a67b021e f415b90edf63 2024-03-22T05:37:31.780 INFO:teuthology.orchestra.run.smithi005.stdout:prometheus.a smithi186 *:9095 running (2m) 2m ago 17m 42.3M - 2.33.4 514e6a882f6e 546020b39d24 2024-03-22T05:37:31.780 INFO:teuthology.orchestra.run.smithi005.stdout:rgw.foo.smithi005.exdhqu smithi005 *:8000 running (16m) 2m ago 16m 76.4M - 17.2.0 e1d6a67b021e 6bb27fe11553 2024-03-22T05:37:31.781 INFO:teuthology.orchestra.run.smithi005.stdout:rgw.foo.smithi186.mcgodk smithi186 *:8000 running (16m) 2m ago 16m 75.0M - 17.2.0 e1d6a67b021e 19b5c20a34f7 2024-03-22T05:37:32.452 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph versions' 2024-03-22T05:37:32.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:32 smithi005 ceph-mon[24793]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:37:32.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:32 smithi005 ceph-mon[24793]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:37:32.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:32 smithi005 ceph-mon[24793]: from='client.25219 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:37:32.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:32 smithi005 ceph-mon[29523]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:37:32.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:32 smithi005 ceph-mon[29523]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:37:32.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:32 smithi005 ceph-mon[29523]: from='client.25219 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:37:33.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:32 smithi186 ceph-mon[25720]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:37:33.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:32 smithi186 ceph-mon[25720]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:37:33.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:32 smithi186 ceph-mon[25720]: from='client.25219 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:37:34.624 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-03-22T05:37:34.624 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": { 2024-03-22T05:37:34.624 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-03-22T05:37:34.624 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:37:34.624 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr": { 2024-03-22T05:37:34.624 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev)": 2 2024-03-22T05:37:34.625 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:37:34.625 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": { 2024-03-22T05:37:34.625 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-22T05:37:34.625 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:37:34.625 INFO:teuthology.orchestra.run.smithi005.stdout: "mds": {}, 2024-03-22T05:37:34.625 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw": { 2024-03-22T05:37:34.625 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-22T05:37:34.625 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:37:34.625 INFO:teuthology.orchestra.run.smithi005.stdout: "overall": { 2024-03-22T05:37:34.625 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-03-22T05:37:34.625 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev)": 2 2024-03-22T05:37:34.625 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-03-22T05:37:34.625 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-03-22T05:37:34.626 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:34 smithi005 ceph-mon[24793]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:37:34.626 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:34 smithi005 ceph-mon[29523]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:37:34.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:34 smithi186 ceph-mon[25720]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:37:35.302 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph -s' 2024-03-22T05:37:35.619 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:35 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/1122706572' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:37:35.619 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:35 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/1122706572' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:37:35.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:35 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/1122706572' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:37:36.635 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:36 smithi005 ceph-mon[24793]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:37:36.635 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:36 smithi005 ceph-mon[29523]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:37:36.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:36 smithi186 ceph-mon[25720]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:37:37.202 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:37:36 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:37:36] "GET /metrics HTTP/1.1" 200 37973 "" "Prometheus/2.33.4" 2024-03-22T05:37:37.383 INFO:teuthology.orchestra.run.smithi005.stdout: cluster: 2024-03-22T05:37:37.383 INFO:teuthology.orchestra.run.smithi005.stdout: id: f59f36e2-e80a-11ee-95cd-87774f69a715 2024-03-22T05:37:37.383 INFO:teuthology.orchestra.run.smithi005.stdout: health: HEALTH_WARN 2024-03-22T05:37:37.383 INFO:teuthology.orchestra.run.smithi005.stdout: 1 failed cephadm daemon(s) 2024-03-22T05:37:37.383 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-22T05:37:37.383 INFO:teuthology.orchestra.run.smithi005.stdout: services: 2024-03-22T05:37:37.383 INFO:teuthology.orchestra.run.smithi005.stdout: mon: 3 daemons, quorum a,c,b (age 21m) 2024-03-22T05:37:37.383 INFO:teuthology.orchestra.run.smithi005.stdout: mgr: y(active, since 3m), standbys: x 2024-03-22T05:37:37.383 INFO:teuthology.orchestra.run.smithi005.stdout: osd: 8 osds: 8 up (since 18m), 8 in (since 18m) 2024-03-22T05:37:37.383 INFO:teuthology.orchestra.run.smithi005.stdout: rgw: 2 daemons active (2 hosts, 1 zones) 2024-03-22T05:37:37.383 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-22T05:37:37.384 INFO:teuthology.orchestra.run.smithi005.stdout: data: 2024-03-22T05:37:37.384 INFO:teuthology.orchestra.run.smithi005.stdout: pools: 6 pools, 161 pgs 2024-03-22T05:37:37.384 INFO:teuthology.orchestra.run.smithi005.stdout: objects: 241 objects, 585 KiB 2024-03-22T05:37:37.384 INFO:teuthology.orchestra.run.smithi005.stdout: usage: 95 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:37:37.384 INFO:teuthology.orchestra.run.smithi005.stdout: pgs: 161 active+clean 2024-03-22T05:37:37.384 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-22T05:37:37.384 INFO:teuthology.orchestra.run.smithi005.stdout: io: 2024-03-22T05:37:37.384 INFO:teuthology.orchestra.run.smithi005.stdout: client: 853 B/s rd, 0 op/s rd, 0 op/s wr 2024-03-22T05:37:37.384 INFO:teuthology.orchestra.run.smithi005.stdout: 2024-03-22T05:37:37.701 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:37 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/321557327' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-22T05:37:37.701 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:37 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/321557327' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-22T05:37:37.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:37 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/321557327' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-22T05:37:38.033 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph health detail' 2024-03-22T05:37:38.654 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:38 smithi005 ceph-mon[29523]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:37:38.655 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:38 smithi005 ceph-mon[24793]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:37:38.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:38 smithi186 ceph-mon[25720]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:37:40.139 INFO:teuthology.orchestra.run.smithi005.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-22T05:37:40.139 INFO:teuthology.orchestra.run.smithi005.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-22T05:37:40.139 INFO:teuthology.orchestra.run.smithi005.stdout: daemon grafana.a on smithi186 is in error state 2024-03-22T05:37:40.750 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:40 smithi005 ceph-mon[29523]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:37:40.750 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:40 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/2526029098' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T05:37:40.750 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:40 smithi005 ceph-mon[24793]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:37:40.750 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:40 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/2526029098' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T05:37:40.773 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph versions | jq -e '"'"'.mgr | length == 1'"'"'' 2024-03-22T05:37:41.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:40 smithi186 ceph-mon[25720]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:37:41.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:40 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/2526029098' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T05:37:42.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:42 smithi005 ceph-mon[29523]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:37:42.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:42 smithi005 ceph-mon[29523]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:37:42.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:42 smithi005 ceph-mon[29523]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:37:42.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:42 smithi005 ceph-mon[24793]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:37:42.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:42 smithi005 ceph-mon[24793]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:37:42.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:42 smithi005 ceph-mon[24793]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:37:42.831 INFO:teuthology.orchestra.run.smithi005.stdout:true 2024-03-22T05:37:43.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:42 smithi186 ceph-mon[25720]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:37:43.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:42 smithi186 ceph-mon[25720]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:37:43.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:42 smithi186 ceph-mon[25720]: from='mgr.25111 172.21.15.5:0/4099684715' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:37:43.481 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph mgr fail' 2024-03-22T05:37:43.794 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:43 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/1977365643' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:37:43.794 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:43 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/1977365643' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:37:44.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:43 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/1977365643' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:37:44.893 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:44 smithi005 ceph-mon[24793]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:37:44.894 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:44 smithi005 ceph-mon[29523]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:37:45.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:44 smithi186 ceph-mon[25720]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:37:46.828 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:37:46 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:37:46.585+0000 7f84966f9700 -1 mgr handle_mgr_map I was active but no longer am 2024-03-22T05:37:46.828 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:37:46 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ignoring --setuser ceph since I am not root 2024-03-22T05:37:46.828 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:37:46 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ignoring --setgroup ceph since I am not root 2024-03-22T05:37:46.828 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:37:46 smithi005 ceph-mgr[63233]: -- 172.21.15.5:0/3481369614 <== mon.0 v2:172.21.15.5:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x564809b085a0 con 0x564809adcc00 2024-03-22T05:37:46.829 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:46 smithi005 ceph-mon[24793]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:37:46.829 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:46 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/780766215' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-03-22T05:37:46.829 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:46 smithi005 ceph-mon[24793]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-03-22T05:37:46.829 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:46 smithi005 ceph-mon[24793]: osdmap e103: 8 total, 8 up, 8 in 2024-03-22T05:37:46.829 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:46 smithi005 ceph-mon[29523]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:37:46.829 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:46 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/780766215' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-03-22T05:37:46.829 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:46 smithi005 ceph-mon[29523]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-03-22T05:37:46.829 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:46 smithi005 ceph-mon[29523]: osdmap e103: 8 total, 8 up, 8 in 2024-03-22T05:37:46.859 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:46 smithi186 ceph-mon[25720]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:37:46.859 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:46 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/780766215' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-03-22T05:37:46.859 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:46 smithi186 ceph-mon[25720]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-03-22T05:37:46.859 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:46 smithi186 ceph-mon[25720]: osdmap e103: 8 total, 8 up, 8 in 2024-03-22T05:37:46.859 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:37:46 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: [22/Mar/2024:05:37:46] ENGINE Bus STOPPING 2024-03-22T05:37:46.859 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:37:46 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: [22/Mar/2024:05:37:46] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-22T05:37:46.859 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:37:46 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: [22/Mar/2024:05:37:46] ENGINE Bus STOPPED 2024-03-22T05:37:46.859 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:37:46 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: [22/Mar/2024:05:37:46] ENGINE Bus STARTING 2024-03-22T05:37:46.860 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:37:46 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: [22/Mar/2024:05:37:46] ENGINE Serving on http://:::9283 2024-03-22T05:37:46.860 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:37:46 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: [22/Mar/2024:05:37:46] ENGINE Bus STARTED 2024-03-22T05:37:47.331 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'sleep 180' 2024-03-22T05:37:47.452 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:37:47 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:37:47.192+0000 7f48f64e9200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-22T05:37:47.452 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:37:47 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:37:47.293+0000 7f48f64e9200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-22T05:37:47.859 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:47 smithi186 ceph-mon[25720]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-03-22T05:37:47.859 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:47 smithi186 ceph-mon[25720]: mgrmap e34: x(active, starting, since 0.991983s) 2024-03-22T05:37:47.859 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:47 smithi186 ceph-mon[25720]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-22T05:37:47.859 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:47 smithi186 ceph-mon[25720]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T05:37:47.859 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:47 smithi186 ceph-mon[25720]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T05:37:47.859 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:47 smithi186 ceph-mon[25720]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-22T05:37:47.859 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:47 smithi186 ceph-mon[25720]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T05:37:47.860 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:47 smithi186 ceph-mon[25720]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T05:37:47.860 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:47 smithi186 ceph-mon[25720]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T05:37:47.860 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:47 smithi186 ceph-mon[25720]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T05:37:47.860 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:47 smithi186 ceph-mon[25720]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T05:37:47.860 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:47 smithi186 ceph-mon[25720]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T05:37:47.860 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:47 smithi186 ceph-mon[25720]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-22T05:37:47.860 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:47 smithi186 ceph-mon[25720]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-22T05:37:47.860 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:47 smithi186 ceph-mon[25720]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-22T05:37:47.860 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:47 smithi186 ceph-mon[25720]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-22T05:37:47.860 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:47 smithi186 ceph-mon[25720]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-22T05:37:47.860 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:47 smithi186 ceph-mon[25720]: Manager daemon x is now available 2024-03-22T05:37:47.860 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:47 smithi186 ceph-mon[25720]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:37:47.860 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:47 smithi186 ceph-mon[25720]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:37:47.860 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:47 smithi186 ceph-mon[25720]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-22T05:37:47.861 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:47 smithi186 ceph-mon[25720]: from='mgr.34976 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-22T05:37:47.861 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:47 smithi186 ceph-mon[25720]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-22T05:37:47.861 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:47 smithi186 ceph-mon[25720]: from='mgr.34976 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-22T05:37:47.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:47 smithi005 ceph-mon[24793]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-03-22T05:37:47.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:47 smithi005 ceph-mon[24793]: mgrmap e34: x(active, starting, since 0.991983s) 2024-03-22T05:37:47.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:47 smithi005 ceph-mon[24793]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-22T05:37:47.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:47 smithi005 ceph-mon[24793]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T05:37:47.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:47 smithi005 ceph-mon[24793]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T05:37:47.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:47 smithi005 ceph-mon[24793]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-22T05:37:47.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:47 smithi005 ceph-mon[24793]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T05:37:47.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:47 smithi005 ceph-mon[24793]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T05:37:47.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:47 smithi005 ceph-mon[24793]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T05:37:47.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:47 smithi005 ceph-mon[24793]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T05:37:47.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:47 smithi005 ceph-mon[24793]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T05:37:47.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:47 smithi005 ceph-mon[24793]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T05:37:47.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:47 smithi005 ceph-mon[24793]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-22T05:37:47.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:47 smithi005 ceph-mon[24793]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-22T05:37:47.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:47 smithi005 ceph-mon[24793]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-22T05:37:47.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:47 smithi005 ceph-mon[24793]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-22T05:37:47.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:47 smithi005 ceph-mon[24793]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-22T05:37:47.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:47 smithi005 ceph-mon[24793]: Manager daemon x is now available 2024-03-22T05:37:47.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:47 smithi005 ceph-mon[24793]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:37:47.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:47 smithi005 ceph-mon[24793]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:37:47.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:47 smithi005 ceph-mon[24793]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-22T05:37:47.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:47 smithi005 ceph-mon[24793]: from='mgr.34976 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-22T05:37:47.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:47 smithi005 ceph-mon[24793]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-22T05:37:47.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:47 smithi005 ceph-mon[24793]: from='mgr.34976 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-22T05:37:47.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:47 smithi005 ceph-mon[29523]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-03-22T05:37:47.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:47 smithi005 ceph-mon[29523]: mgrmap e34: x(active, starting, since 0.991983s) 2024-03-22T05:37:47.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:47 smithi005 ceph-mon[29523]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-22T05:37:47.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:47 smithi005 ceph-mon[29523]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T05:37:47.955 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:47 smithi005 ceph-mon[29523]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T05:37:47.955 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:47 smithi005 ceph-mon[29523]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-22T05:37:47.955 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:47 smithi005 ceph-mon[29523]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T05:37:47.955 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:47 smithi005 ceph-mon[29523]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T05:37:47.955 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:47 smithi005 ceph-mon[29523]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T05:37:47.955 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:47 smithi005 ceph-mon[29523]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T05:37:47.955 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:47 smithi005 ceph-mon[29523]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T05:37:47.955 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:47 smithi005 ceph-mon[29523]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T05:37:47.955 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:47 smithi005 ceph-mon[29523]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-22T05:37:47.955 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:47 smithi005 ceph-mon[29523]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-22T05:37:47.955 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:47 smithi005 ceph-mon[29523]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-22T05:37:47.955 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:47 smithi005 ceph-mon[29523]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-22T05:37:47.955 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:47 smithi005 ceph-mon[29523]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-22T05:37:47.955 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:47 smithi005 ceph-mon[29523]: Manager daemon x is now available 2024-03-22T05:37:47.956 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:47 smithi005 ceph-mon[29523]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:37:47.956 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:47 smithi005 ceph-mon[29523]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:37:47.956 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:47 smithi005 ceph-mon[29523]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-22T05:37:47.956 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:47 smithi005 ceph-mon[29523]: from='mgr.34976 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-22T05:37:47.956 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:47 smithi005 ceph-mon[29523]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-22T05:37:47.956 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:47 smithi005 ceph-mon[29523]: from='mgr.34976 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-22T05:37:48.366 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:37:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:37:48.018+0000 7f48f64e9200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-22T05:37:48.366 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:37:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:37:48.116+0000 7f48f64e9200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-22T05:37:48.366 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:37:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:37:48.265+0000 7f48f64e9200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-22T05:37:48.366 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:37:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:37:48.366+0000 7f48f64e9200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-22T05:37:48.644 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:37:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:37:48.457+0000 7f48f64e9200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-22T05:37:48.644 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:48 smithi005 ceph-mon[24793]: [22/Mar/2024:05:37:47] ENGINE Bus STARTING 2024-03-22T05:37:48.644 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:48 smithi005 ceph-mon[24793]: [22/Mar/2024:05:37:47] ENGINE Serving on http://172.21.15.186:8765 2024-03-22T05:37:48.644 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:48 smithi005 ceph-mon[24793]: [22/Mar/2024:05:37:47] ENGINE Serving on https://172.21.15.186:7150 2024-03-22T05:37:48.644 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:48 smithi005 ceph-mon[24793]: [22/Mar/2024:05:37:47] ENGINE Bus STARTED 2024-03-22T05:37:48.645 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:48 smithi005 ceph-mon[24793]: mgrmap e35: x(active, since 1.99376s) 2024-03-22T05:37:48.645 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:48 smithi005 ceph-mon[24793]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:37:48.645 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:48 smithi005 ceph-mon[29523]: [22/Mar/2024:05:37:47] ENGINE Bus STARTING 2024-03-22T05:37:48.645 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:48 smithi005 ceph-mon[29523]: [22/Mar/2024:05:37:47] ENGINE Serving on http://172.21.15.186:8765 2024-03-22T05:37:48.645 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:48 smithi005 ceph-mon[29523]: [22/Mar/2024:05:37:47] ENGINE Serving on https://172.21.15.186:7150 2024-03-22T05:37:48.645 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:48 smithi005 ceph-mon[29523]: [22/Mar/2024:05:37:47] ENGINE Bus STARTED 2024-03-22T05:37:48.645 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:48 smithi005 ceph-mon[29523]: mgrmap e35: x(active, since 1.99376s) 2024-03-22T05:37:48.645 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:48 smithi005 ceph-mon[29523]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:37:48.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:48 smithi186 ceph-mon[25720]: [22/Mar/2024:05:37:47] ENGINE Bus STARTING 2024-03-22T05:37:48.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:48 smithi186 ceph-mon[25720]: [22/Mar/2024:05:37:47] ENGINE Serving on http://172.21.15.186:8765 2024-03-22T05:37:48.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:48 smithi186 ceph-mon[25720]: [22/Mar/2024:05:37:47] ENGINE Serving on https://172.21.15.186:7150 2024-03-22T05:37:48.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:48 smithi186 ceph-mon[25720]: [22/Mar/2024:05:37:47] ENGINE Bus STARTED 2024-03-22T05:37:48.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:48 smithi186 ceph-mon[25720]: mgrmap e35: x(active, since 1.99376s) 2024-03-22T05:37:48.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:48 smithi186 ceph-mon[25720]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:37:48.946 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:37:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:37:48.735+0000 7f48f64e9200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-22T05:37:48.947 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:37:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:37:48.848+0000 7f48f64e9200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-22T05:37:48.947 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:37:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:37:48.947+0000 7f48f64e9200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-22T05:37:49.202 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:37:49 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:37:49.045+0000 7f48f64e9200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-22T05:37:49.558 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:37:49 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:37:49.226+0000 7f48f64e9200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-22T05:37:49.893 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:49 smithi005 ceph-mon[24793]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-22T05:37:49.893 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:49 smithi005 ceph-mon[24793]: Cluster is now healthy 2024-03-22T05:37:49.893 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:49 smithi005 ceph-mon[24793]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:37:49.893 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:49 smithi005 ceph-mon[29523]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-22T05:37:49.893 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:49 smithi005 ceph-mon[29523]: Cluster is now healthy 2024-03-22T05:37:49.893 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:49 smithi005 ceph-mon[29523]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:37:49.940 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:49 smithi186 ceph-mon[25720]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-22T05:37:49.940 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:49 smithi186 ceph-mon[25720]: Cluster is now healthy 2024-03-22T05:37:49.940 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:49 smithi186 ceph-mon[25720]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:37:50.602 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:37:50 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:37:50.265+0000 7f48f64e9200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-22T05:37:50.602 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:37:50 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:37:50.411+0000 7f48f64e9200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-22T05:37:50.877 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:50 smithi005 ceph-mon[24793]: mgrmap e36: x(active, since 4s) 2024-03-22T05:37:50.878 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:37:50 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:37:50.783+0000 7f48f64e9200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-22T05:37:50.878 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:50 smithi005 ceph-mon[29523]: mgrmap e36: x(active, since 4s) 2024-03-22T05:37:51.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:50 smithi186 ceph-mon[25720]: mgrmap e36: x(active, since 4s) 2024-03-22T05:37:51.200 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:37:50 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:37:50.877+0000 7f48f64e9200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-22T05:37:51.200 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:37:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:37:51.088+0000 7f48f64e9200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-22T05:37:51.452 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:37:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:37:51.284+0000 7f48f64e9200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-22T05:37:51.850 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:37:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:37:51.594+0000 7f48f64e9200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-22T05:37:51.850 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:51 smithi005 ceph-mon[29523]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:37:51.850 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:51 smithi005 ceph-mon[29523]: from='mgr.34976 ' entity='mgr.x' 2024-03-22T05:37:51.850 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:51 smithi005 ceph-mon[29523]: from='mgr.34976 ' entity='mgr.x' 2024-03-22T05:37:51.851 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:51 smithi005 ceph-mon[24793]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:37:51.851 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:51 smithi005 ceph-mon[24793]: from='mgr.34976 ' entity='mgr.x' 2024-03-22T05:37:51.851 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:51 smithi005 ceph-mon[24793]: from='mgr.34976 ' entity='mgr.x' 2024-03-22T05:37:52.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:51 smithi186 ceph-mon[25720]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:37:52.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:51 smithi186 ceph-mon[25720]: from='mgr.34976 ' entity='mgr.x' 2024-03-22T05:37:52.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:51 smithi186 ceph-mon[25720]: from='mgr.34976 ' entity='mgr.x' 2024-03-22T05:37:52.112 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:37:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:37:51.864+0000 7f48f64e9200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-22T05:37:52.112 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:37:52 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:37:52.112+0000 7f48f64e9200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-22T05:37:52.638 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:52 smithi005 ceph-mon[24793]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:37:52.638 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:52 smithi005 ceph-mon[24793]: from='mgr.34976 ' entity='mgr.x' 2024-03-22T05:37:52.638 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:52 smithi005 ceph-mon[24793]: from='mgr.34976 ' entity='mgr.x' 2024-03-22T05:37:52.638 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:52 smithi005 ceph-mon[29523]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:37:52.638 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:52 smithi005 ceph-mon[29523]: from='mgr.34976 ' entity='mgr.x' 2024-03-22T05:37:52.638 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:52 smithi005 ceph-mon[29523]: from='mgr.34976 ' entity='mgr.x' 2024-03-22T05:37:53.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:52 smithi186 ceph-mon[25720]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:37:53.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:52 smithi186 ceph-mon[25720]: from='mgr.34976 ' entity='mgr.x' 2024-03-22T05:37:53.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:52 smithi186 ceph-mon[25720]: from='mgr.34976 ' entity='mgr.x' 2024-03-22T05:37:53.452 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:37:53 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:37:53.020+0000 7f48f64e9200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-22T05:37:53.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:53 smithi005 ceph-mon[29523]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 24 op/s 2024-03-22T05:37:53.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:53 smithi005 ceph-mon[29523]: from='mgr.34976 ' entity='mgr.x' 2024-03-22T05:37:53.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:53 smithi005 ceph-mon[29523]: from='mgr.34976 ' entity='mgr.x' 2024-03-22T05:37:53.952 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:37:53 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:37:53.630+0000 7f48f64e9200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-22T05:37:53.952 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:37:53 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:37:53.736+0000 7f48f64e9200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-22T05:37:53.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:53 smithi005 ceph-mon[24793]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 24 op/s 2024-03-22T05:37:53.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:53 smithi005 ceph-mon[24793]: from='mgr.34976 ' entity='mgr.x' 2024-03-22T05:37:53.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:53 smithi005 ceph-mon[24793]: from='mgr.34976 ' entity='mgr.x' 2024-03-22T05:37:54.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:53 smithi186 ceph-mon[25720]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 24 op/s 2024-03-22T05:37:54.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:53 smithi186 ceph-mon[25720]: from='mgr.34976 ' entity='mgr.x' 2024-03-22T05:37:54.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:53 smithi186 ceph-mon[25720]: from='mgr.34976 ' entity='mgr.x' 2024-03-22T05:37:54.452 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:37:53 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:37:53.975+0000 7f48f64e9200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-22T05:37:54.452 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:37:53 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: [22/Mar/2024:05:37:53] ENGINE Bus STARTING 2024-03-22T05:37:54.452 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:37:54 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: CherryPy Checker: 2024-03-22T05:37:54.452 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:37:54 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: The Application mounted at '' has an empty config. 2024-03-22T05:37:54.452 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:37:54 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:37:54.452 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:37:54 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: [22/Mar/2024:05:37:54] ENGINE Serving on http://:::9283 2024-03-22T05:37:54.452 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:37:54 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: [22/Mar/2024:05:37:54] ENGINE Bus STARTED 2024-03-22T05:37:54.944 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:54 smithi186 ceph-mon[25720]: Standby manager daemon y started 2024-03-22T05:37:54.944 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:54 smithi186 ceph-mon[25720]: from='mgr.? 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-22T05:37:54.944 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:54 smithi186 ceph-mon[25720]: from='mgr.? 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-22T05:37:54.945 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:54 smithi186 ceph-mon[25720]: from='mgr.? 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-22T05:37:54.945 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:54 smithi186 ceph-mon[25720]: from='mgr.? 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-22T05:37:54.945 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:54 smithi186 ceph-mon[25720]: from='mgr.34976 ' entity='mgr.x' 2024-03-22T05:37:54.945 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:54 smithi186 ceph-mon[25720]: from='mgr.34976 ' entity='mgr.x' 2024-03-22T05:37:54.945 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:54 smithi186 ceph-mon[25720]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:37:54.945 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:54 smithi186 ceph-mon[25720]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:37:54.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:54 smithi005 ceph-mon[24793]: Standby manager daemon y started 2024-03-22T05:37:54.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:54 smithi005 ceph-mon[24793]: from='mgr.? 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-22T05:37:54.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:54 smithi005 ceph-mon[24793]: from='mgr.? 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-22T05:37:54.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:54 smithi005 ceph-mon[24793]: from='mgr.? 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-22T05:37:54.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:54 smithi005 ceph-mon[24793]: from='mgr.? 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-22T05:37:54.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:54 smithi005 ceph-mon[24793]: from='mgr.34976 ' entity='mgr.x' 2024-03-22T05:37:54.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:54 smithi005 ceph-mon[24793]: from='mgr.34976 ' entity='mgr.x' 2024-03-22T05:37:54.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:54 smithi005 ceph-mon[24793]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:37:54.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:54 smithi005 ceph-mon[24793]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:37:54.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:54 smithi005 ceph-mon[29523]: Standby manager daemon y started 2024-03-22T05:37:54.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:54 smithi005 ceph-mon[29523]: from='mgr.? 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-22T05:37:54.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:54 smithi005 ceph-mon[29523]: from='mgr.? 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-22T05:37:54.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:54 smithi005 ceph-mon[29523]: from='mgr.? 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-22T05:37:54.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:54 smithi005 ceph-mon[29523]: from='mgr.? 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-22T05:37:54.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:54 smithi005 ceph-mon[29523]: from='mgr.34976 ' entity='mgr.x' 2024-03-22T05:37:54.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:54 smithi005 ceph-mon[29523]: from='mgr.34976 ' entity='mgr.x' 2024-03-22T05:37:54.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:54 smithi005 ceph-mon[29523]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:37:54.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:54 smithi005 ceph-mon[29523]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:37:55.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:55 smithi005 ceph-mon[24793]: Updating smithi005:/etc/ceph/ceph.conf 2024-03-22T05:37:55.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:55 smithi005 ceph-mon[24793]: Updating smithi186:/etc/ceph/ceph.conf 2024-03-22T05:37:55.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:55 smithi005 ceph-mon[24793]: Updating smithi186:/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T05:37:55.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:55 smithi005 ceph-mon[24793]: Updating smithi005:/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T05:37:55.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:55 smithi005 ceph-mon[24793]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-22T05:37:55.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:55 smithi005 ceph-mon[24793]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-22T05:37:55.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:55 smithi005 ceph-mon[24793]: mgrmap e37: x(active, since 9s), standbys: y 2024-03-22T05:37:55.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:55 smithi005 ceph-mon[24793]: from='mgr.34976 ' entity='mgr.x' 2024-03-22T05:37:55.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:55 smithi005 ceph-mon[24793]: from='mgr.34976 ' entity='mgr.x' 2024-03-22T05:37:55.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:55 smithi005 ceph-mon[24793]: from='mgr.34976 ' entity='mgr.x' 2024-03-22T05:37:55.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:55 smithi005 ceph-mon[24793]: from='mgr.34976 ' entity='mgr.x' 2024-03-22T05:37:55.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:55 smithi005 ceph-mon[24793]: from='mgr.34976 ' entity='mgr.x' 2024-03-22T05:37:55.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:55 smithi005 ceph-mon[29523]: Updating smithi005:/etc/ceph/ceph.conf 2024-03-22T05:37:55.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:55 smithi005 ceph-mon[29523]: Updating smithi186:/etc/ceph/ceph.conf 2024-03-22T05:37:55.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:55 smithi005 ceph-mon[29523]: Updating smithi186:/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T05:37:55.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:55 smithi005 ceph-mon[29523]: Updating smithi005:/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T05:37:55.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:55 smithi005 ceph-mon[29523]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-22T05:37:55.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:55 smithi005 ceph-mon[29523]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-22T05:37:55.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:55 smithi005 ceph-mon[29523]: mgrmap e37: x(active, since 9s), standbys: y 2024-03-22T05:37:55.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:55 smithi005 ceph-mon[29523]: from='mgr.34976 ' entity='mgr.x' 2024-03-22T05:37:55.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:55 smithi005 ceph-mon[29523]: from='mgr.34976 ' entity='mgr.x' 2024-03-22T05:37:55.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:55 smithi005 ceph-mon[29523]: from='mgr.34976 ' entity='mgr.x' 2024-03-22T05:37:55.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:55 smithi005 ceph-mon[29523]: from='mgr.34976 ' entity='mgr.x' 2024-03-22T05:37:55.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:55 smithi005 ceph-mon[29523]: from='mgr.34976 ' entity='mgr.x' 2024-03-22T05:37:56.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:55 smithi186 ceph-mon[25720]: Updating smithi005:/etc/ceph/ceph.conf 2024-03-22T05:37:56.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:55 smithi186 ceph-mon[25720]: Updating smithi186:/etc/ceph/ceph.conf 2024-03-22T05:37:56.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:55 smithi186 ceph-mon[25720]: Updating smithi186:/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T05:37:56.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:55 smithi186 ceph-mon[25720]: Updating smithi005:/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T05:37:56.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:55 smithi186 ceph-mon[25720]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-22T05:37:56.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:55 smithi186 ceph-mon[25720]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-22T05:37:56.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:55 smithi186 ceph-mon[25720]: mgrmap e37: x(active, since 9s), standbys: y 2024-03-22T05:37:56.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:55 smithi186 ceph-mon[25720]: from='mgr.34976 ' entity='mgr.x' 2024-03-22T05:37:56.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:55 smithi186 ceph-mon[25720]: from='mgr.34976 ' entity='mgr.x' 2024-03-22T05:37:56.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:55 smithi186 ceph-mon[25720]: from='mgr.34976 ' entity='mgr.x' 2024-03-22T05:37:56.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:55 smithi186 ceph-mon[25720]: from='mgr.34976 ' entity='mgr.x' 2024-03-22T05:37:56.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:55 smithi186 ceph-mon[25720]: from='mgr.34976 ' entity='mgr.x' 2024-03-22T05:37:56.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:56 smithi005 ceph-mon[24793]: Updating smithi186:/etc/ceph/ceph.client.admin.keyring 2024-03-22T05:37:56.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:56 smithi005 ceph-mon[24793]: Updating smithi005:/etc/ceph/ceph.client.admin.keyring 2024-03-22T05:37:56.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:56 smithi005 ceph-mon[24793]: Updating smithi186:/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-22T05:37:56.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:56 smithi005 ceph-mon[24793]: Updating smithi005:/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-22T05:37:56.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:56 smithi005 ceph-mon[24793]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 15 op/s 2024-03-22T05:37:56.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:56 smithi005 ceph-mon[24793]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-22T05:37:56.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:56 smithi005 ceph-mon[24793]: Health check failed: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-22T05:37:56.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:56 smithi005 ceph-mon[29523]: Updating smithi186:/etc/ceph/ceph.client.admin.keyring 2024-03-22T05:37:56.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:56 smithi005 ceph-mon[29523]: Updating smithi005:/etc/ceph/ceph.client.admin.keyring 2024-03-22T05:37:56.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:56 smithi005 ceph-mon[29523]: Updating smithi186:/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-22T05:37:56.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:56 smithi005 ceph-mon[29523]: Updating smithi005:/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-22T05:37:56.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:56 smithi005 ceph-mon[29523]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 15 op/s 2024-03-22T05:37:56.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:56 smithi005 ceph-mon[29523]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-22T05:37:56.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:56 smithi005 ceph-mon[29523]: Health check failed: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-22T05:37:57.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:56 smithi186 ceph-mon[25720]: Updating smithi186:/etc/ceph/ceph.client.admin.keyring 2024-03-22T05:37:57.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:56 smithi186 ceph-mon[25720]: Updating smithi005:/etc/ceph/ceph.client.admin.keyring 2024-03-22T05:37:57.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:56 smithi186 ceph-mon[25720]: Updating smithi186:/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-22T05:37:57.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:56 smithi186 ceph-mon[25720]: Updating smithi005:/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-22T05:37:57.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:56 smithi186 ceph-mon[25720]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 15 op/s 2024-03-22T05:37:57.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:56 smithi186 ceph-mon[25720]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-22T05:37:57.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:56 smithi186 ceph-mon[25720]: Health check failed: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-22T05:37:57.452 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:37:56 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:37:56] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-22T05:37:57.917 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:57 smithi186 ceph-mon[25720]: Reconfiguring daemon prometheus.a on smithi186 2024-03-22T05:37:57.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:57 smithi005 ceph-mon[24793]: Reconfiguring daemon prometheus.a on smithi186 2024-03-22T05:37:57.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:57 smithi005 ceph-mon[29523]: Reconfiguring daemon prometheus.a on smithi186 2024-03-22T05:37:58.671 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:58 smithi186 ceph-mon[25720]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-22T05:37:58.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:58 smithi005 ceph-mon[24793]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-22T05:37:58.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:58 smithi005 ceph-mon[29523]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-22T05:37:58.959 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:37:58 smithi186 systemd[1]: Stopping Ceph prometheus.a for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:37:58.959 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:37:58 smithi186 bash[57856]: Error: no container with name or ID "ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus.a" found: no such container 2024-03-22T05:37:59.293 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:37:58 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[55010]: ts=2024-03-22T05:37:58.959Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-22T05:37:59.294 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:37:58 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[55010]: ts=2024-03-22T05:37:58.959Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-03-22T05:37:59.294 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:37:58 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[55010]: ts=2024-03-22T05:37:58.959Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-03-22T05:37:59.294 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:37:58 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[55010]: ts=2024-03-22T05:37:58.959Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-03-22T05:37:59.294 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:37:58 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[55010]: ts=2024-03-22T05:37:58.959Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-03-22T05:37:59.294 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:37:58 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[55010]: ts=2024-03-22T05:37:58.959Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-03-22T05:37:59.294 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:37:58 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[55010]: ts=2024-03-22T05:37:58.959Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-22T05:37:59.294 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:37:58 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[55010]: ts=2024-03-22T05:37:58.959Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-03-22T05:37:59.294 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:37:58 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[55010]: ts=2024-03-22T05:37:58.959Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-03-22T05:37:59.294 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:37:58 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[55010]: ts=2024-03-22T05:37:58.960Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-03-22T05:37:59.294 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:37:58 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[55010]: ts=2024-03-22T05:37:58.960Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-03-22T05:37:59.294 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:37:58 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[55010]: ts=2024-03-22T05:37:58.960Z caller=main.go:1066 level=info msg="See you next time!" 2024-03-22T05:37:59.294 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:37:58 smithi186 podman[57868]: 2024-03-22 05:37:58.969648541 +0000 UTC m=+0.125225228 container died 546020b39d245819eb217e9db65507e56522d1ae4801d13358ab95ee5127c25b (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-22T05:37:59.543 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:37:59 smithi186 podman[57868]: 2024-03-22 05:37:59.392722166 +0000 UTC m=+0.548298851 container cleanup 546020b39d245819eb217e9db65507e56522d1ae4801d13358ab95ee5127c25b (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-22T05:37:59.544 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:37:59 smithi186 bash[57868]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a 2024-03-22T05:37:59.544 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:37:59 smithi186 bash[57894]: Error: no container with name or ID "ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus.a" found: no such container 2024-03-22T05:37:59.869 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:37:59 smithi186 ceph-mon[25720]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-22T05:37:59.869 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:37:59 smithi186 podman[57882]: 2024-03-22 05:37:59.602418797 +0000 UTC m=+0.635422145 container remove 546020b39d245819eb217e9db65507e56522d1ae4801d13358ab95ee5127c25b (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-22T05:37:59.869 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:37:59 smithi186 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-03-22T05:37:59.869 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:37:59 smithi186 systemd[1]: Stopped Ceph prometheus.a for f59f36e2-e80a-11ee-95cd-87774f69a715. 2024-03-22T05:37:59.869 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:37:59 smithi186 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@prometheus.a.service: Consumed 1.412s CPU time. 2024-03-22T05:37:59.870 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:37:59 smithi186 systemd[1]: Starting Ceph prometheus.a for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:37:59.870 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:37:59 smithi186 podman[57965]: 2024-03-22 05:37:59.734497662 +0000 UTC m=+0.020589356 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-03-22T05:37:59.870 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:37:59 smithi186 podman[57965]: 2024-03-22 05:37:59.8702325 +0000 UTC m=+0.156324186 container create bf952ea306b42abec9d95c50ed050f5e7d5e18c2315fc833f132f42bd3816a74 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-22T05:37:59.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:37:59 smithi005 ceph-mon[24793]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-22T05:37:59.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:37:59 smithi005 ceph-mon[29523]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-22T05:38:00.122 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:38:00 smithi186 podman[57965]: 2024-03-22 05:38:00.097482 +0000 UTC m=+0.383573687 container init bf952ea306b42abec9d95c50ed050f5e7d5e18c2315fc833f132f42bd3816a74 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-22T05:38:00.122 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:38:00 smithi186 podman[57965]: 2024-03-22 05:38:00.100107061 +0000 UTC m=+0.386198749 container start bf952ea306b42abec9d95c50ed050f5e7d5e18c2315fc833f132f42bd3816a74 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-22T05:38:00.544 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:38:00 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[57980]: ts=2024-03-22T05:38:00.123Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-03-22T05:38:00.544 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:38:00 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[57980]: ts=2024-03-22T05:38:00.123Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-03-22T05:38:00.544 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:38:00 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[57980]: ts=2024-03-22T05:38:00.123Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-03-22T05:38:00.544 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:38:00 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[57980]: ts=2024-03-22T05:38:00.123Z caller=main.go:518 level=info host_details="(Linux 5.14.0-430.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024 x86_64 smithi186 (none))" 2024-03-22T05:38:00.544 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:38:00 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[57980]: ts=2024-03-22T05:38:00.123Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-22T05:38:00.544 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:38:00 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[57980]: ts=2024-03-22T05:38:00.123Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-22T05:38:00.545 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:38:00 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[57980]: ts=2024-03-22T05:38:00.125Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-03-22T05:38:00.545 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:38:00 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[57980]: ts=2024-03-22T05:38:00.125Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-03-22T05:38:00.545 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:38:00 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[57980]: ts=2024-03-22T05:38:00.126Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-03-22T05:38:00.545 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:38:00 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[57980]: ts=2024-03-22T05:38:00.127Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-22T05:38:00.545 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:38:00 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[57980]: ts=2024-03-22T05:38:00.127Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.618µs 2024-03-22T05:38:00.545 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:38:00 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[57980]: ts=2024-03-22T05:38:00.127Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-22T05:38:00.545 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:38:00 smithi186 bash[57965]: bf952ea306b42abec9d95c50ed050f5e7d5e18c2315fc833f132f42bd3816a74 2024-03-22T05:38:00.545 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:38:00 smithi186 systemd[1]: Started Ceph prometheus.a for f59f36e2-e80a-11ee-95cd-87774f69a715. 2024-03-22T05:38:00.545 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:38:00 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[57980]: ts=2024-03-22T05:38:00.343Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=4 2024-03-22T05:38:00.842 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:38:00 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[57980]: ts=2024-03-22T05:38:00.582Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=4 2024-03-22T05:38:00.842 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:38:00 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[57980]: ts=2024-03-22T05:38:00.789Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=4 2024-03-22T05:38:00.842 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:38:00 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[57980]: ts=2024-03-22T05:38:00.843Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=4 2024-03-22T05:38:01.101 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:38:00 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[57980]: ts=2024-03-22T05:38:00.843Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=4 maxSegment=4 2024-03-22T05:38:01.101 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:38:00 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[57980]: ts=2024-03-22T05:38:00.843Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=35.784µs wal_replay_duration=715.589071ms total_replay_duration=715.641674ms 2024-03-22T05:38:01.101 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:38:00 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[57980]: ts=2024-03-22T05:38:00.848Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-03-22T05:38:01.101 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:38:00 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[57980]: ts=2024-03-22T05:38:00.848Z caller=main.go:947 level=info msg="TSDB started" 2024-03-22T05:38:01.101 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:38:00 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[57980]: ts=2024-03-22T05:38:00.848Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-22T05:38:01.101 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:38:00 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[57980]: ts=2024-03-22T05:38:00.873Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=25.1865ms db_storage=949ns remote_storage=1.63µs web_handler=574ns query_engine=736ns scrape=386.905µs scrape_sd=102.746µs notify=11.001µs notify_sd=9.11µs rules=24.325696ms 2024-03-22T05:38:01.101 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:38:00 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[57980]: ts=2024-03-22T05:38:00.873Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-03-22T05:38:01.376 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:38:01 smithi186 ceph-mon[25720]: from='mgr.34976 ' entity='mgr.x' 2024-03-22T05:38:01.376 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:38:01 smithi186 ceph-mon[25720]: from='mgr.34976 ' entity='mgr.x' 2024-03-22T05:38:01.376 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:38:01 smithi186 ceph-mon[25720]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-22T05:38:01.376 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:38:01 smithi186 ceph-mon[25720]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-22T05:38:01.376 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:38:01 smithi186 ceph-mon[25720]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:38:01.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:38:01 smithi005 ceph-mon[24793]: from='mgr.34976 ' entity='mgr.x' 2024-03-22T05:38:01.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:38:01 smithi005 ceph-mon[24793]: from='mgr.34976 ' entity='mgr.x' 2024-03-22T05:38:01.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:38:01 smithi005 ceph-mon[24793]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-22T05:38:01.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:38:01 smithi005 ceph-mon[24793]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-22T05:38:01.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:38:01 smithi005 ceph-mon[24793]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:38:01.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:38:01 smithi005 ceph-mon[29523]: from='mgr.34976 ' entity='mgr.x' 2024-03-22T05:38:01.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:38:01 smithi005 ceph-mon[29523]: from='mgr.34976 ' entity='mgr.x' 2024-03-22T05:38:01.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:38:01 smithi005 ceph-mon[29523]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-22T05:38:01.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:38:01 smithi005 ceph-mon[29523]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-22T05:38:01.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:38:01 smithi005 ceph-mon[29523]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:38:02.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:38:02 smithi186 ceph-mon[25720]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:38:02.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:38:02 smithi186 ceph-mon[25720]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-22T05:38:02.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:38:02 smithi186 ceph-mon[25720]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:38:02.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:38:02 smithi005 ceph-mon[24793]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:38:02.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:38:02 smithi005 ceph-mon[24793]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-22T05:38:02.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:38:02 smithi005 ceph-mon[24793]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:38:02.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:38:02 smithi005 ceph-mon[29523]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:38:02.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:38:02 smithi005 ceph-mon[29523]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-22T05:38:02.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:38:02 smithi005 ceph-mon[29523]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:38:04.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:38:04 smithi005 ceph-mon[24793]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 950 B/s rd, 0 op/s 2024-03-22T05:38:04.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:38:04 smithi005 ceph-mon[24793]: from='mgr.34976 ' entity='mgr.x' 2024-03-22T05:38:04.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:38:04 smithi005 ceph-mon[24793]: from='mgr.34976 ' entity='mgr.x' 2024-03-22T05:38:04.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:38:04 smithi005 ceph-mon[24793]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:38:04.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:38:04 smithi005 ceph-mon[24793]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:38:04.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:38:04 smithi005 ceph-mon[24793]: from='mgr.34976 ' entity='mgr.x' 2024-03-22T05:38:04.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:38:04 smithi005 ceph-mon[29523]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 950 B/s rd, 0 op/s 2024-03-22T05:38:04.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:38:04 smithi005 ceph-mon[29523]: from='mgr.34976 ' entity='mgr.x' 2024-03-22T05:38:04.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:38:04 smithi005 ceph-mon[29523]: from='mgr.34976 ' entity='mgr.x' 2024-03-22T05:38:04.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:38:04 smithi005 ceph-mon[29523]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:38:04.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:38:04 smithi005 ceph-mon[29523]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:38:04.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:38:04 smithi005 ceph-mon[29523]: from='mgr.34976 ' entity='mgr.x' 2024-03-22T05:38:04.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:38:04 smithi186 ceph-mon[25720]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 950 B/s rd, 0 op/s 2024-03-22T05:38:04.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:38:04 smithi186 ceph-mon[25720]: from='mgr.34976 ' entity='mgr.x' 2024-03-22T05:38:04.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:38:04 smithi186 ceph-mon[25720]: from='mgr.34976 ' entity='mgr.x' 2024-03-22T05:38:04.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:38:04 smithi186 ceph-mon[25720]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:38:04.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:38:04 smithi186 ceph-mon[25720]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:38:04.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:38:04 smithi186 ceph-mon[25720]: from='mgr.34976 ' entity='mgr.x' 2024-03-22T05:38:05.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:38:05 smithi005 ceph-mon[29523]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-22T05:38:05.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:38:05 smithi005 ceph-mon[29523]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:38:05.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:38:05 smithi005 ceph-mon[24793]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-22T05:38:05.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:38:05 smithi005 ceph-mon[24793]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:38:05.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:38:05 smithi186 ceph-mon[25720]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-22T05:38:05.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:38:05 smithi186 ceph-mon[25720]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:38:07.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:38:07 smithi005 ceph-mon[24793]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:38:07.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:38:07 smithi005 ceph-mon[29523]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:38:07.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:38:07 smithi186 ceph-mon[25720]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:38:09.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:38:09 smithi005 ceph-mon[24793]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:38:09.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:38:09 smithi005 ceph-mon[29523]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:38:09.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:38:09 smithi186 ceph-mon[25720]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:38:11.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:38:11 smithi005 ceph-mon[24793]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:38:11.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:38:11 smithi005 ceph-mon[29523]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:38:11.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:38:11 smithi186 ceph-mon[25720]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:38:11.794 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:38:11 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:38:11] "GET /metrics HTTP/1.1" 200 37977 "" "Prometheus/2.33.4" 2024-03-22T05:38:12.698 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:38:12 smithi005 ceph-mon[29523]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:38:12.698 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:38:12 smithi005 ceph-mon[24793]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:38:12.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:38:12 smithi186 ceph-mon[25720]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:38:13.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:38:13 smithi005 ceph-mon[24793]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 583 B/s rd, 0 op/s 2024-03-22T05:38:13.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:38:13 smithi005 ceph-mon[29523]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 583 B/s rd, 0 op/s 2024-03-22T05:38:13.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:38:13 smithi186 ceph-mon[25720]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 583 B/s rd, 0 op/s 2024-03-22T05:38:15.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:38:15 smithi005 ceph-mon[24793]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:38:15.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:38:15 smithi005 ceph-mon[29523]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:38:15.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:38:15 smithi186 ceph-mon[25720]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:38:17.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:38:17 smithi005 ceph-mon[24793]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:38:17.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:38:17 smithi005 ceph-mon[24793]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:38:17.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:38:17 smithi005 ceph-mon[29523]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:38:17.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:38:17 smithi005 ceph-mon[29523]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:38:17.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:38:17 smithi186 ceph-mon[25720]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:38:17.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:38:17 smithi186 ceph-mon[25720]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:38:19.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:38:19 smithi005 ceph-mon[29523]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:38:19.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:38:19 smithi005 ceph-mon[24793]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:38:19.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:38:19 smithi186 ceph-mon[25720]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:38:21.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:38:21 smithi005 ceph-mon[29523]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:38:21.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:38:21 smithi005 ceph-mon[24793]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:38:21.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:38:21 smithi186 ceph-mon[25720]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:38:21.793 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:38:21 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:38:21] "GET /metrics HTTP/1.1" 200 37976 "" "Prometheus/2.33.4" 2024-03-22T05:38:22.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:38:22 smithi005 ceph-mon[29523]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:38:22.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:38:22 smithi005 ceph-mon[24793]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:38:22.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:38:22 smithi186 ceph-mon[25720]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:38:23.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:38:23 smithi005 ceph-mon[29523]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:38:23.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:38:23 smithi005 ceph-mon[24793]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:38:23.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:38:23 smithi186 ceph-mon[25720]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:38:25.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:38:25 smithi005 ceph-mon[24793]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:38:25.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:38:25 smithi005 ceph-mon[29523]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:38:25.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:38:25 smithi186 ceph-mon[25720]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:38:27.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:38:27 smithi005 ceph-mon[24793]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:38:27.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:38:27 smithi005 ceph-mon[29523]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:38:27.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:38:27 smithi186 ceph-mon[25720]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:38:29.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:38:29 smithi005 ceph-mon[24793]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:38:29.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:38:29 smithi005 ceph-mon[29523]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:38:29.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:38:29 smithi186 ceph-mon[25720]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:38:31.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:38:31 smithi005 ceph-mon[24793]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:38:31.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:38:31 smithi005 ceph-mon[29523]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:38:31.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:38:31 smithi186 ceph-mon[25720]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:38:31.793 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:38:31 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:38:31] "GET /metrics HTTP/1.1" 200 37976 "" "Prometheus/2.33.4" 2024-03-22T05:38:32.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:38:32 smithi005 ceph-mon[24793]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:38:32.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:38:32 smithi005 ceph-mon[24793]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:38:32.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:38:32 smithi005 ceph-mon[29523]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:38:32.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:38:32 smithi005 ceph-mon[29523]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:38:32.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:38:32 smithi186 ceph-mon[25720]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:38:32.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:38:32 smithi186 ceph-mon[25720]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:38:33.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:38:33 smithi005 ceph-mon[24793]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:38:33.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:38:33 smithi005 ceph-mon[29523]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:38:33.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:38:33 smithi186 ceph-mon[25720]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:38:35.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:38:35 smithi005 ceph-mon[24793]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:38:35.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:38:35 smithi005 ceph-mon[29523]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:38:35.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:38:35 smithi186 ceph-mon[25720]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:38:37.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:38:37 smithi005 ceph-mon[24793]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:38:37.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:38:37 smithi005 ceph-mon[29523]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:38:37.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:38:37 smithi186 ceph-mon[25720]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:38:39.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:38:39 smithi005 ceph-mon[24793]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:38:39.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:38:39 smithi005 ceph-mon[29523]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:38:39.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:38:39 smithi186 ceph-mon[25720]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:38:41.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:38:41 smithi005 ceph-mon[24793]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:38:41.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:38:41 smithi005 ceph-mon[29523]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:38:41.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:38:41 smithi186 ceph-mon[25720]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:38:41.793 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:38:41 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:38:41] "GET /metrics HTTP/1.1" 200 37973 "" "Prometheus/2.33.4" 2024-03-22T05:38:42.689 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:38:42 smithi005 ceph-mon[24793]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:38:42.689 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:38:42 smithi005 ceph-mon[29523]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:38:42.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:38:42 smithi186 ceph-mon[25720]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:38:43.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:38:43 smithi005 ceph-mon[24793]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:38:43.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:38:43 smithi005 ceph-mon[29523]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:38:43.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:38:43 smithi186 ceph-mon[25720]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:38:45.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:38:45 smithi005 ceph-mon[24793]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:38:45.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:38:45 smithi005 ceph-mon[29523]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:38:45.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:38:45 smithi186 ceph-mon[25720]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:38:47.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:38:47 smithi005 ceph-mon[29523]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:38:47.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:38:47 smithi005 ceph-mon[29523]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:38:47.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:38:47 smithi005 ceph-mon[24793]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:38:47.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:38:47 smithi005 ceph-mon[24793]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:38:47.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:38:47 smithi186 ceph-mon[25720]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:38:47.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:38:47 smithi186 ceph-mon[25720]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:38:49.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:38:49 smithi005 ceph-mon[24793]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:38:49.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:38:49 smithi005 ceph-mon[29523]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:38:49.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:38:49 smithi186 ceph-mon[25720]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:38:51.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:38:51 smithi005 ceph-mon[24793]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:38:51.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:38:51 smithi005 ceph-mon[29523]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:38:51.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:38:51 smithi186 ceph-mon[25720]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:38:51.793 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:38:51 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:38:51] "GET /metrics HTTP/1.1" 200 37974 "" "Prometheus/2.33.4" 2024-03-22T05:38:52.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:38:52 smithi005 ceph-mon[29523]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:38:52.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:38:52 smithi005 ceph-mon[24793]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:38:52.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:38:52 smithi186 ceph-mon[25720]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:38:53.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:38:53 smithi005 ceph-mon[24793]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:38:53.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:38:53 smithi005 ceph-mon[29523]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:38:53.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:38:53 smithi186 ceph-mon[25720]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:38:55.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:38:55 smithi005 ceph-mon[24793]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:38:55.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:38:55 smithi005 ceph-mon[29523]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:38:55.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:38:55 smithi186 ceph-mon[25720]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:38:57.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:38:57 smithi005 ceph-mon[24793]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:38:57.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:38:57 smithi005 ceph-mon[29523]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:38:57.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:38:57 smithi186 ceph-mon[25720]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:38:59.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:38:59 smithi005 ceph-mon[24793]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:38:59.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:38:59 smithi005 ceph-mon[29523]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:38:59.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:38:59 smithi186 ceph-mon[25720]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:39:01.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:39:01 smithi005 ceph-mon[24793]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:39:01.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:39:01 smithi005 ceph-mon[29523]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:39:01.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:39:01 smithi186 ceph-mon[25720]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:39:01.794 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:39:01 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:39:01] "GET /metrics HTTP/1.1" 200 37974 "" "Prometheus/2.33.4" 2024-03-22T05:39:02.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:39:02 smithi005 ceph-mon[24793]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:39:02.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:39:02 smithi005 ceph-mon[24793]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:39:02.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:39:02 smithi005 ceph-mon[29523]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:39:02.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:39:02 smithi005 ceph-mon[29523]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:39:02.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:39:02 smithi186 ceph-mon[25720]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:39:02.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:39:02 smithi186 ceph-mon[25720]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:39:03.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:39:03 smithi005 ceph-mon[29523]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:39:03.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:39:03 smithi005 ceph-mon[24793]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:39:03.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:39:03 smithi186 ceph-mon[25720]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:39:04.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:39:04 smithi186 ceph-mon[25720]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:39:04.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:39:04 smithi005 ceph-mon[24793]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:39:04.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:39:04 smithi005 ceph-mon[29523]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:39:05.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:39:05 smithi186 ceph-mon[25720]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:39:05.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:39:05 smithi186 ceph-mon[25720]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:39:05.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:39:05 smithi186 ceph-mon[25720]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:39:05.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:39:05 smithi186 ceph-mon[25720]: from='mgr.34976 ' entity='mgr.x' 2024-03-22T05:39:05.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:39:05 smithi005 ceph-mon[24793]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:39:05.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:39:05 smithi005 ceph-mon[24793]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:39:05.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:39:05 smithi005 ceph-mon[24793]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:39:05.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:39:05 smithi005 ceph-mon[24793]: from='mgr.34976 ' entity='mgr.x' 2024-03-22T05:39:05.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:39:05 smithi005 ceph-mon[29523]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:39:05.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:39:05 smithi005 ceph-mon[29523]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:39:05.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:39:05 smithi005 ceph-mon[29523]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:39:05.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:39:05 smithi005 ceph-mon[29523]: from='mgr.34976 ' entity='mgr.x' 2024-03-22T05:39:06.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:39:06 smithi186 ceph-mon[25720]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 951 B/s rd, 0 op/s 2024-03-22T05:39:06.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:39:06 smithi186 ceph-mon[25720]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:39:06.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:39:06 smithi005 ceph-mon[24793]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 951 B/s rd, 0 op/s 2024-03-22T05:39:06.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:39:06 smithi005 ceph-mon[24793]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:39:06.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:39:06 smithi005 ceph-mon[29523]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 951 B/s rd, 0 op/s 2024-03-22T05:39:06.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:39:06 smithi005 ceph-mon[29523]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:39:08.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:39:08 smithi186 ceph-mon[25720]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 584 B/s rd, 0 op/s 2024-03-22T05:39:08.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:39:08 smithi005 ceph-mon[24793]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 584 B/s rd, 0 op/s 2024-03-22T05:39:08.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:39:08 smithi005 ceph-mon[29523]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 584 B/s rd, 0 op/s 2024-03-22T05:39:10.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:39:10 smithi186 ceph-mon[25720]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:39:10.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:39:10 smithi005 ceph-mon[24793]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:39:10.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:39:10 smithi005 ceph-mon[29523]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:39:11.794 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:39:11 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:39:11] "GET /metrics HTTP/1.1" 200 37970 "" "Prometheus/2.33.4" 2024-03-22T05:39:12.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:39:12 smithi186 ceph-mon[25720]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:39:12.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:39:12 smithi186 ceph-mon[25720]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:39:12.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:39:12 smithi005 ceph-mon[29523]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:39:12.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:39:12 smithi005 ceph-mon[29523]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:39:12.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:39:12 smithi005 ceph-mon[24793]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:39:12.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:39:12 smithi005 ceph-mon[24793]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:39:14.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:39:14 smithi186 ceph-mon[25720]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:39:14.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:39:14 smithi005 ceph-mon[24793]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:39:14.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:39:14 smithi005 ceph-mon[29523]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:39:16.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:39:16 smithi186 ceph-mon[25720]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:39:16.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:39:16 smithi005 ceph-mon[24793]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:39:16.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:39:16 smithi005 ceph-mon[29523]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:39:17.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:39:17 smithi186 ceph-mon[25720]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:39:17.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:39:17 smithi005 ceph-mon[24793]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:39:17.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:39:17 smithi005 ceph-mon[29523]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:39:18.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:39:18 smithi186 ceph-mon[25720]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:39:18.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:39:18 smithi005 ceph-mon[29523]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:39:18.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:39:18 smithi005 ceph-mon[24793]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:39:20.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:39:20 smithi186 ceph-mon[25720]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:39:20.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:39:20 smithi005 ceph-mon[29523]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:39:20.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:39:20 smithi005 ceph-mon[24793]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:39:21.793 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:39:21 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:39:21] "GET /metrics HTTP/1.1" 200 37976 "" "Prometheus/2.33.4" 2024-03-22T05:39:22.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:39:22 smithi186 ceph-mon[25720]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:39:22.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:39:22 smithi186 ceph-mon[25720]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:39:22.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:39:22 smithi005 ceph-mon[29523]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:39:22.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:39:22 smithi005 ceph-mon[29523]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:39:22.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:39:22 smithi005 ceph-mon[24793]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:39:22.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:39:22 smithi005 ceph-mon[24793]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:39:24.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:39:24 smithi186 ceph-mon[25720]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:39:24.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:39:24 smithi005 ceph-mon[29523]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:39:24.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:39:24 smithi005 ceph-mon[24793]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:39:26.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:39:26 smithi186 ceph-mon[25720]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:39:26.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:39:26 smithi005 ceph-mon[29523]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:39:26.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:39:26 smithi005 ceph-mon[24793]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:39:28.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:39:28 smithi186 ceph-mon[25720]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:39:28.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:39:28 smithi005 ceph-mon[29523]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:39:28.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:39:28 smithi005 ceph-mon[24793]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:39:30.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:39:30 smithi186 ceph-mon[25720]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:39:30.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:39:30 smithi005 ceph-mon[29523]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:39:30.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:39:30 smithi005 ceph-mon[24793]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:39:31.793 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:39:31 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:39:31] "GET /metrics HTTP/1.1" 200 37976 "" "Prometheus/2.33.4" 2024-03-22T05:39:32.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:39:32 smithi186 ceph-mon[25720]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:39:32.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:39:32 smithi186 ceph-mon[25720]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:39:32.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:39:32 smithi186 ceph-mon[25720]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:39:32.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:39:32 smithi005 ceph-mon[29523]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:39:32.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:39:32 smithi005 ceph-mon[29523]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:39:32.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:39:32 smithi005 ceph-mon[29523]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:39:32.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:39:32 smithi005 ceph-mon[24793]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:39:32.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:39:32 smithi005 ceph-mon[24793]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:39:32.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:39:32 smithi005 ceph-mon[24793]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:39:34.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:39:34 smithi186 ceph-mon[25720]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:39:34.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:39:34 smithi005 ceph-mon[24793]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:39:34.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:39:34 smithi005 ceph-mon[29523]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:39:36.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:39:36 smithi186 ceph-mon[25720]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:39:36.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:39:36 smithi005 ceph-mon[24793]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:39:36.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:39:36 smithi005 ceph-mon[29523]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:39:38.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:39:38 smithi186 ceph-mon[25720]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:39:38.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:39:38 smithi005 ceph-mon[24793]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:39:38.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:39:38 smithi005 ceph-mon[29523]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:39:40.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:39:40 smithi186 ceph-mon[25720]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:39:40.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:39:40 smithi005 ceph-mon[24793]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:39:40.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:39:40 smithi005 ceph-mon[29523]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:39:41.793 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:39:41 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:39:41] "GET /metrics HTTP/1.1" 200 37973 "" "Prometheus/2.33.4" 2024-03-22T05:39:42.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:39:42 smithi186 ceph-mon[25720]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:39:42.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:39:42 smithi186 ceph-mon[25720]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:39:42.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:39:42 smithi005 ceph-mon[24793]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:39:42.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:39:42 smithi005 ceph-mon[24793]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:39:42.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:39:42 smithi005 ceph-mon[29523]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:39:42.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:39:42 smithi005 ceph-mon[29523]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:39:44.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:39:44 smithi186 ceph-mon[25720]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:39:44.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:39:44 smithi005 ceph-mon[24793]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:39:44.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:39:44 smithi005 ceph-mon[29523]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:39:46.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:39:46 smithi186 ceph-mon[25720]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-22T05:39:46.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:39:46 smithi005 ceph-mon[24793]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-22T05:39:46.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:39:46 smithi005 ceph-mon[29523]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-22T05:39:47.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:39:47 smithi186 ceph-mon[25720]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:39:47.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:39:47 smithi005 ceph-mon[24793]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:39:47.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:39:47 smithi005 ceph-mon[29523]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:39:48.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:39:48 smithi186 ceph-mon[25720]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-22T05:39:48.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:39:48 smithi005 ceph-mon[24793]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-22T05:39:48.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:39:48 smithi005 ceph-mon[29523]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-22T05:39:50.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:39:50 smithi186 ceph-mon[25720]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:39:50.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:39:50 smithi005 ceph-mon[24793]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:39:50.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:39:50 smithi005 ceph-mon[29523]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:39:51.794 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:39:51 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:39:51] "GET /metrics HTTP/1.1" 200 37973 "" "Prometheus/2.33.4" 2024-03-22T05:39:52.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:39:52 smithi186 ceph-mon[25720]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-22T05:39:52.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:39:52 smithi186 ceph-mon[25720]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:39:52.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:39:52 smithi005 ceph-mon[24793]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-22T05:39:52.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:39:52 smithi005 ceph-mon[24793]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:39:52.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:39:52 smithi005 ceph-mon[29523]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-22T05:39:52.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:39:52 smithi005 ceph-mon[29523]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:39:54.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:39:54 smithi186 ceph-mon[25720]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:39:54.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:39:54 smithi005 ceph-mon[24793]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:39:54.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:39:54 smithi005 ceph-mon[29523]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:39:56.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:39:56 smithi186 ceph-mon[25720]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:39:56.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:39:56 smithi005 ceph-mon[24793]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:39:56.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:39:56 smithi005 ceph-mon[29523]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:39:58.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:39:58 smithi186 ceph-mon[25720]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:39:58.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:39:58 smithi005 ceph-mon[24793]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:39:58.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:39:58 smithi005 ceph-mon[29523]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:40:00.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:40:00 smithi186 ceph-mon[25720]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:40:00.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:40:00 smithi186 ceph-mon[25720]: Health detail: HEALTH_WARN 2 failed cephadm daemon(s) 2024-03-22T05:40:00.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:40:00 smithi186 ceph-mon[25720]: [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2024-03-22T05:40:00.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:40:00 smithi186 ceph-mon[25720]: daemon iscsi.foo.smithi005.mcdwbp on smithi005 is in error state 2024-03-22T05:40:00.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:40:00 smithi186 ceph-mon[25720]: daemon grafana.a on smithi186 is in error state 2024-03-22T05:40:00.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:40:00 smithi005 ceph-mon[29523]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:40:00.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:40:00 smithi005 ceph-mon[29523]: Health detail: HEALTH_WARN 2 failed cephadm daemon(s) 2024-03-22T05:40:00.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:40:00 smithi005 ceph-mon[29523]: [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2024-03-22T05:40:00.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:40:00 smithi005 ceph-mon[29523]: daemon iscsi.foo.smithi005.mcdwbp on smithi005 is in error state 2024-03-22T05:40:00.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:40:00 smithi005 ceph-mon[29523]: daemon grafana.a on smithi186 is in error state 2024-03-22T05:40:00.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:40:00 smithi005 ceph-mon[24793]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:40:00.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:40:00 smithi005 ceph-mon[24793]: Health detail: HEALTH_WARN 2 failed cephadm daemon(s) 2024-03-22T05:40:00.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:40:00 smithi005 ceph-mon[24793]: [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s) 2024-03-22T05:40:00.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:40:00 smithi005 ceph-mon[24793]: daemon iscsi.foo.smithi005.mcdwbp on smithi005 is in error state 2024-03-22T05:40:00.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:40:00 smithi005 ceph-mon[24793]: daemon grafana.a on smithi186 is in error state 2024-03-22T05:40:01.793 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:40:01 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:40:01] "GET /metrics HTTP/1.1" 200 37973 "" "Prometheus/2.33.4" 2024-03-22T05:40:02.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:40:02 smithi005 ceph-mon[29523]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:40:02.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:40:02 smithi005 ceph-mon[29523]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:40:02.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:40:02 smithi005 ceph-mon[29523]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:40:02.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:40:02 smithi005 ceph-mon[24793]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:40:02.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:40:02 smithi005 ceph-mon[24793]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:40:02.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:40:02 smithi005 ceph-mon[24793]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:40:03.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:40:02 smithi186 ceph-mon[25720]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:40:03.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:40:02 smithi186 ceph-mon[25720]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:40:03.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:40:02 smithi186 ceph-mon[25720]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:40:04.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:40:04 smithi005 ceph-mon[29523]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:40:04.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:40:04 smithi005 ceph-mon[24793]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:40:05.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:40:04 smithi186 ceph-mon[25720]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:40:05.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:40:05 smithi005 ceph-mon[24793]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:40:05.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:40:05 smithi005 ceph-mon[29523]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:40:06.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:40:05 smithi186 ceph-mon[25720]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:40:06.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:40:06 smithi005 ceph-mon[24793]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:40:06.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:40:06 smithi005 ceph-mon[24793]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:40:06.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:40:06 smithi005 ceph-mon[24793]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:40:06.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:40:06 smithi005 ceph-mon[24793]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 958 B/s rd, 0 op/s 2024-03-22T05:40:06.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:40:06 smithi005 ceph-mon[24793]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:40:06.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:40:06 smithi005 ceph-mon[24793]: from='mgr.34976 ' entity='mgr.x' 2024-03-22T05:40:06.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:40:06 smithi005 ceph-mon[29523]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:40:06.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:40:06 smithi005 ceph-mon[29523]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:40:06.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:40:06 smithi005 ceph-mon[29523]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:40:06.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:40:06 smithi005 ceph-mon[29523]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 958 B/s rd, 0 op/s 2024-03-22T05:40:06.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:40:06 smithi005 ceph-mon[29523]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:40:06.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:40:06 smithi005 ceph-mon[29523]: from='mgr.34976 ' entity='mgr.x' 2024-03-22T05:40:07.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:40:06 smithi186 ceph-mon[25720]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:40:07.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:40:06 smithi186 ceph-mon[25720]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:40:07.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:40:06 smithi186 ceph-mon[25720]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:40:07.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:40:06 smithi186 ceph-mon[25720]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 958 B/s rd, 0 op/s 2024-03-22T05:40:07.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:40:06 smithi186 ceph-mon[25720]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:40:07.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:40:06 smithi186 ceph-mon[25720]: from='mgr.34976 ' entity='mgr.x' 2024-03-22T05:40:07.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:40:07 smithi005 ceph-mon[24793]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:40:07.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:40:07 smithi005 ceph-mon[29523]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:40:08.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:40:07 smithi186 ceph-mon[25720]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:40:09.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:40:09 smithi005 ceph-mon[24793]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:40:09.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:40:09 smithi005 ceph-mon[29523]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:40:10.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:40:09 smithi186 ceph-mon[25720]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:40:11.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:40:11 smithi186 ceph-mon[25720]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:40:11.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:40:11 smithi186 ceph-mon[25720]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 589 B/s rd, 0 op/s 2024-03-22T05:40:11.794 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:40:11 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:40:11] "GET /metrics HTTP/1.1" 200 37986 "" "Prometheus/2.33.4" 2024-03-22T05:40:11.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:40:11 smithi005 ceph-mon[24793]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:40:11.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:40:11 smithi005 ceph-mon[24793]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 589 B/s rd, 0 op/s 2024-03-22T05:40:11.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:40:11 smithi005 ceph-mon[29523]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:40:11.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:40:11 smithi005 ceph-mon[29523]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 589 B/s rd, 0 op/s 2024-03-22T05:40:14.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:40:14 smithi005 ceph-mon[24793]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:40:14.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:40:14 smithi005 ceph-mon[29523]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:40:14.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:40:14 smithi186 ceph-mon[25720]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:40:15.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:40:15 smithi005 ceph-mon[24793]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:40:15.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:40:15 smithi005 ceph-mon[29523]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:40:16.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:40:15 smithi186 ceph-mon[25720]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:40:17.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:40:16 smithi186 ceph-mon[25720]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:40:17.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:40:16 smithi005 ceph-mon[24793]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:40:17.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:40:16 smithi005 ceph-mon[29523]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:40:18.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:40:17 smithi186 ceph-mon[25720]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:40:18.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:40:17 smithi005 ceph-mon[24793]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:40:18.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:40:17 smithi005 ceph-mon[29523]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:40:19.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:40:19 smithi005 ceph-mon[24793]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:40:19.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:40:19 smithi005 ceph-mon[29523]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:40:20.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:40:19 smithi186 ceph-mon[25720]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:40:21.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:40:21 smithi005 ceph-mon[24793]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:40:21.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:40:21 smithi005 ceph-mon[24793]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:40:21.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:40:21 smithi005 ceph-mon[29523]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:40:21.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:40:21 smithi005 ceph-mon[29523]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:40:21.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:40:21 smithi186 ceph-mon[25720]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:40:21.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:40:21 smithi186 ceph-mon[25720]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:40:21.794 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:40:21 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:40:21] "GET /metrics HTTP/1.1" 200 37986 "" "Prometheus/2.33.4" 2024-03-22T05:40:24.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:40:24 smithi005 ceph-mon[24793]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:40:24.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:40:24 smithi005 ceph-mon[29523]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:40:24.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:40:24 smithi186 ceph-mon[25720]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:40:25.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:40:25 smithi005 ceph-mon[24793]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:40:25.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:40:25 smithi005 ceph-mon[29523]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:40:26.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:40:25 smithi186 ceph-mon[25720]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:40:27.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:40:27 smithi005 ceph-mon[24793]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:40:27.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:40:27 smithi005 ceph-mon[29523]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:40:28.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:40:27 smithi186 ceph-mon[25720]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:40:29.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:40:29 smithi005 ceph-mon[24793]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:40:29.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:40:29 smithi005 ceph-mon[29523]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:40:30.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:40:29 smithi186 ceph-mon[25720]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:40:31.794 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:40:31 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:40:31] "GET /metrics HTTP/1.1" 200 37986 "" "Prometheus/2.33.4" 2024-03-22T05:40:31.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:40:31 smithi186 ceph-mon[25720]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:40:31.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:40:31 smithi186 ceph-mon[25720]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:40:31.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:40:31 smithi186 ceph-mon[25720]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:40:31.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:40:31 smithi005 ceph-mon[24793]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:40:31.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:40:31 smithi005 ceph-mon[24793]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:40:31.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:40:31 smithi005 ceph-mon[24793]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:40:31.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:40:31 smithi005 ceph-mon[29523]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:40:31.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:40:31 smithi005 ceph-mon[29523]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:40:31.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:40:31 smithi005 ceph-mon[29523]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:40:34.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:40:34 smithi005 ceph-mon[24793]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:40:34.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:40:34 smithi005 ceph-mon[29523]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:40:34.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:40:34 smithi186 ceph-mon[25720]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:40:35.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:40:35 smithi005 ceph-mon[29523]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:40:35.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:40:35 smithi005 ceph-mon[24793]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:40:36.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:40:35 smithi186 ceph-mon[25720]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:40:37.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:40:37 smithi005 ceph-mon[24793]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:40:37.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:40:37 smithi005 ceph-mon[29523]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:40:38.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:40:37 smithi186 ceph-mon[25720]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:40:39.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:40:39 smithi005 ceph-mon[29523]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:40:39.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:40:39 smithi005 ceph-mon[24793]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:40:40.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:40:39 smithi186 ceph-mon[25720]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:40:41.754 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:40:41 smithi005 ceph-mon[29523]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:40:41.754 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:40:41 smithi005 ceph-mon[29523]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:40:41.755 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:40:41 smithi005 ceph-mon[24793]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:40:41.755 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:40:41 smithi005 ceph-mon[24793]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:40:41.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:40:41 smithi186 ceph-mon[25720]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:40:41.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:40:41 smithi186 ceph-mon[25720]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:40:41.794 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:40:41 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:40:41] "GET /metrics HTTP/1.1" 200 37984 "" "Prometheus/2.33.4" 2024-03-22T05:40:44.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:40:44 smithi005 ceph-mon[24793]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:40:44.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:40:44 smithi005 ceph-mon[29523]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:40:44.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:40:44 smithi186 ceph-mon[25720]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:40:45.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:40:45 smithi005 ceph-mon[24793]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:40:45.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:40:45 smithi005 ceph-mon[29523]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:40:46.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:40:45 smithi186 ceph-mon[25720]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:40:46.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:40:46 smithi005 ceph-mon[24793]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:40:46.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:40:46 smithi005 ceph-mon[29523]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:40:47.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:40:46 smithi186 ceph-mon[25720]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:40:48.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:40:47 smithi186 ceph-mon[25720]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:40:48.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:40:47 smithi005 ceph-mon[29523]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:40:48.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:40:47 smithi005 ceph-mon[24793]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:40:49.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:40:49 smithi005 ceph-mon[29523]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:40:49.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:40:49 smithi005 ceph-mon[24793]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:40:50.046 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:40:49 smithi186 ceph-mon[25720]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:40:51.168 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph orch ps' 2024-03-22T05:40:51.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:40:51 smithi186 ceph-mon[25720]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:40:51.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:40:51 smithi186 ceph-mon[25720]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:40:51.794 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:40:51 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:40:51] "GET /metrics HTTP/1.1" 200 37981 "" "Prometheus/2.33.4" 2024-03-22T05:40:51.808 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:40:51 smithi005 ceph-mon[24793]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:40:51.808 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:40:51 smithi005 ceph-mon[24793]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:40:51.808 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:40:51 smithi005 ceph-mon[29523]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:40:51.808 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:40:51 smithi005 ceph-mon[29523]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:40:53.183 INFO:teuthology.orchestra.run.smithi005.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T05:40:53.183 INFO:teuthology.orchestra.run.smithi005.stdout:alertmanager.a smithi005 *:9093,9094 running (12m) 3m ago 20m 22.8M - 0.23.0 ba2b418f427c fd7dc5dbe3e5 2024-03-22T05:40:53.183 INFO:teuthology.orchestra.run.smithi005.stdout:grafana.a smithi186 *:3000 error 2m ago 20m - - 2024-03-22T05:40:53.184 INFO:teuthology.orchestra.run.smithi005.stdout:iscsi.foo.smithi005.mcdwbp smithi005 error 3m ago 19m 55.0M - 3.5 e1d6a67b021e 5215e03bf81b 2024-03-22T05:40:53.184 INFO:teuthology.orchestra.run.smithi005.stdout:mgr.x smithi186 *:8443,9283 running (15m) 2m ago 24m 494M - 19.0.0-2299-gea01217f aa461e7c5e7f 66924e151e7b 2024-03-22T05:40:53.184 INFO:teuthology.orchestra.run.smithi005.stdout:mgr.y smithi005 *:8443,9283,8765 running (9m) 3m ago 26m 72.0M - 19.0.0-2299-gea01217f aa461e7c5e7f 62a813eebc0d 2024-03-22T05:40:53.184 INFO:teuthology.orchestra.run.smithi005.stdout:mon.a smithi005 running (26m) 3m ago 26m 72.2M 2048M 17.2.0 e1d6a67b021e 38e0bba4bb03 2024-03-22T05:40:53.184 INFO:teuthology.orchestra.run.smithi005.stdout:mon.b smithi186 running (25m) 2m ago 25m 49.0M 2048M 17.2.0 e1d6a67b021e 323f53f932f6 2024-03-22T05:40:53.184 INFO:teuthology.orchestra.run.smithi005.stdout:mon.c smithi005 running (25m) 3m ago 25m 53.3M 2048M 17.2.0 e1d6a67b021e 00ec24d38612 2024-03-22T05:40:53.184 INFO:teuthology.orchestra.run.smithi005.stdout:node-exporter.a smithi005 *:9100 running (12m) 3m ago 21m 16.2M - 1.3.1 1dbe0e931976 a9085d6042eb 2024-03-22T05:40:53.184 INFO:teuthology.orchestra.run.smithi005.stdout:node-exporter.b smithi186 *:9100 running (12m) 2m ago 20m 18.4M - 1.3.1 1dbe0e931976 74aa1181ff71 2024-03-22T05:40:53.184 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0 smithi005 running (24m) 3m ago 24m 52.6M 2993M 17.2.0 e1d6a67b021e ef5c7053dd37 2024-03-22T05:40:53.184 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1 smithi005 running (24m) 3m ago 24m 55.1M 2993M 17.2.0 e1d6a67b021e a1e7ee1017b6 2024-03-22T05:40:53.184 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2 smithi005 running (23m) 3m ago 23m 48.1M 2993M 17.2.0 e1d6a67b021e 462ea63ad0e5 2024-03-22T05:40:53.184 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3 smithi005 running (23m) 3m ago 23m 55.1M 2993M 17.2.0 e1d6a67b021e b25432a7239b 2024-03-22T05:40:53.184 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4 smithi186 running (22m) 2m ago 22m 54.0M 3248M 17.2.0 e1d6a67b021e cd963671f340 2024-03-22T05:40:53.184 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5 smithi186 running (22m) 2m ago 22m 50.8M 3248M 17.2.0 e1d6a67b021e f98c87f1a2cd 2024-03-22T05:40:53.185 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6 smithi186 running (22m) 2m ago 22m 50.1M 3248M 17.2.0 e1d6a67b021e 704ac49c68c4 2024-03-22T05:40:53.185 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7 smithi186 running (21m) 2m ago 21m 52.8M 3248M 17.2.0 e1d6a67b021e f415b90edf63 2024-03-22T05:40:53.185 INFO:teuthology.orchestra.run.smithi005.stdout:prometheus.a smithi186 *:9095 running (2m) 2m ago 21m 42.1M - 2.33.4 514e6a882f6e bf952ea306b4 2024-03-22T05:40:53.185 INFO:teuthology.orchestra.run.smithi005.stdout:rgw.foo.smithi005.exdhqu smithi005 *:8000 running (19m) 3m ago 19m 77.5M - 17.2.0 e1d6a67b021e 6bb27fe11553 2024-03-22T05:40:53.185 INFO:teuthology.orchestra.run.smithi005.stdout:rgw.foo.smithi186.mcgodk smithi186 *:8000 running (19m) 2m ago 19m 75.7M - 17.2.0 e1d6a67b021e 19b5c20a34f7 2024-03-22T05:40:53.998 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph orch ls' 2024-03-22T05:40:54.640 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:40:54 smithi005 ceph-mon[24793]: from='client.25285 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:40:54.640 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:40:54 smithi005 ceph-mon[24793]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:40:54.640 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:40:54 smithi005 ceph-mon[29523]: from='client.25285 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:40:54.640 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:40:54 smithi005 ceph-mon[29523]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:40:54.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:40:54 smithi186 ceph-mon[25720]: from='client.25285 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:40:54.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:40:54 smithi186 ceph-mon[25720]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:40:55.722 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:40:55 smithi005 ceph-mon[24793]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:40:55.723 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:40:55 smithi005 ceph-mon[29523]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:40:56.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:40:55 smithi186 ceph-mon[25720]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:40:56.052 INFO:teuthology.orchestra.run.smithi005.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-22T05:40:56.052 INFO:teuthology.orchestra.run.smithi005.stdout:alertmanager ?:9093,9094 1/1 3m ago 21m smithi005=a;count:1 2024-03-22T05:40:56.052 INFO:teuthology.orchestra.run.smithi005.stdout:grafana ?:3000 0/1 2m ago 21m smithi186=a;count:1 2024-03-22T05:40:56.052 INFO:teuthology.orchestra.run.smithi005.stdout:iscsi.foo ?:5000 0/1 3m ago 19m count:1 2024-03-22T05:40:56.052 INFO:teuthology.orchestra.run.smithi005.stdout:mgr 2/2 3m ago 24m smithi005=y;smithi186=x;count:2 2024-03-22T05:40:56.052 INFO:teuthology.orchestra.run.smithi005.stdout:mon 3/3 3m ago 25m smithi005:172.21.15.5=a;smithi005:[v2:172.21.15.5:3301,v1:172.21.15.5:6790]=c;smithi186:172.21.15.186=b;count:3 2024-03-22T05:40:56.052 INFO:teuthology.orchestra.run.smithi005.stdout:node-exporter ?:9100 2/2 3m ago 21m smithi005=a;smithi186=b;count:2 2024-03-22T05:40:56.052 INFO:teuthology.orchestra.run.smithi005.stdout:osd 8 3m ago - 2024-03-22T05:40:56.052 INFO:teuthology.orchestra.run.smithi005.stdout:prometheus ?:9095 1/1 2m ago 21m smithi186=a;count:1 2024-03-22T05:40:56.053 INFO:teuthology.orchestra.run.smithi005.stdout:rgw.foo ?:8000 2/2 3m ago 19m count:2 2024-03-22T05:40:56.768 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph versions' 2024-03-22T05:40:56.794 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:40:56 smithi005 ceph-mon[24793]: from='client.15381 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:40:56.794 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:40:56 smithi005 ceph-mon[29523]: from='client.15381 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:40:57.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:40:56 smithi186 ceph-mon[25720]: from='client.15381 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:40:57.954 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:40:57 smithi186 ceph-mon[25720]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:40:58.007 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:40:57 smithi005 ceph-mon[29523]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:40:58.008 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:40:57 smithi005 ceph-mon[24793]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:40:58.794 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-03-22T05:40:58.795 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": { 2024-03-22T05:40:58.795 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-03-22T05:40:58.795 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:40:58.795 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr": { 2024-03-22T05:40:58.795 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev)": 2 2024-03-22T05:40:58.795 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:40:58.795 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": { 2024-03-22T05:40:58.795 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-22T05:40:58.795 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:40:58.795 INFO:teuthology.orchestra.run.smithi005.stdout: "mds": {}, 2024-03-22T05:40:58.795 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw": { 2024-03-22T05:40:58.795 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-22T05:40:58.796 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:40:58.796 INFO:teuthology.orchestra.run.smithi005.stdout: "overall": { 2024-03-22T05:40:58.796 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-03-22T05:40:58.796 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev)": 2 2024-03-22T05:40:58.796 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-03-22T05:40:58.796 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-03-22T05:40:59.424 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr' 2024-03-22T05:40:59.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:40:59 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/799103202' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:40:59.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:40:59 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/799103202' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:40:59.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:40:59 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/799103202' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:41:00.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:00 smithi005 ceph-mon[24793]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:41:00.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:00 smithi005 ceph-mon[29523]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:41:00.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:00 smithi186 ceph-mon[25720]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:41:01.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:01 smithi186 ceph-mon[25720]: from='client.25303 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:41:01.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:01 smithi186 ceph-mon[25720]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:41:01.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:01 smithi186 ceph-mon[25720]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:41:01.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:01 smithi186 ceph-mon[25720]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:41:01.794 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:41:01 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:41:01] "GET /metrics HTTP/1.1" 200 37981 "" "Prometheus/2.33.4" 2024-03-22T05:41:01.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:01 smithi005 ceph-mon[24793]: from='client.25303 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:41:01.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:01 smithi005 ceph-mon[24793]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:41:01.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:01 smithi005 ceph-mon[24793]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:41:01.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:01 smithi005 ceph-mon[24793]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:41:01.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:01 smithi005 ceph-mon[29523]: from='client.25303 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:41:01.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:01 smithi005 ceph-mon[29523]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:41:01.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:01 smithi005 ceph-mon[29523]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:41:01.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:01 smithi005 ceph-mon[29523]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:41:03.362 INFO:teuthology.orchestra.run.smithi005.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37 2024-03-22T05:41:04.007 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-03-22T05:41:04.618 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:04 smithi005 ceph-mon[24793]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37 2024-03-22T05:41:04.618 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:04 smithi005 ceph-mon[24793]: from='mgr.34976 ' entity='mgr.x' 2024-03-22T05:41:04.618 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:04 smithi005 ceph-mon[24793]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:41:04.618 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:04 smithi005 ceph-mon[24793]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:41:04.618 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:04 smithi005 ceph-mon[24793]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:41:04.618 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:04 smithi005 ceph-mon[24793]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-22T05:41:04.618 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:04 smithi005 ceph-mon[24793]: from='mgr.34976 ' entity='mgr.x' 2024-03-22T05:41:04.618 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:04 smithi005 ceph-mon[24793]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37 2024-03-22T05:41:04.618 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:04 smithi005 ceph-mon[29523]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37 2024-03-22T05:41:04.618 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:04 smithi005 ceph-mon[29523]: from='mgr.34976 ' entity='mgr.x' 2024-03-22T05:41:04.619 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:04 smithi005 ceph-mon[29523]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:41:04.619 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:04 smithi005 ceph-mon[29523]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:41:04.619 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:04 smithi005 ceph-mon[29523]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:41:04.619 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:04 smithi005 ceph-mon[29523]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-22T05:41:04.619 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:04 smithi005 ceph-mon[29523]: from='mgr.34976 ' entity='mgr.x' 2024-03-22T05:41:04.619 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:04 smithi005 ceph-mon[29523]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37 2024-03-22T05:41:04.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:04 smithi186 ceph-mon[25720]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37 2024-03-22T05:41:04.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:04 smithi186 ceph-mon[25720]: from='mgr.34976 ' entity='mgr.x' 2024-03-22T05:41:04.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:04 smithi186 ceph-mon[25720]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:41:04.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:04 smithi186 ceph-mon[25720]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:41:04.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:04 smithi186 ceph-mon[25720]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:41:04.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:04 smithi186 ceph-mon[25720]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-22T05:41:04.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:04 smithi186 ceph-mon[25720]: from='mgr.34976 ' entity='mgr.x' 2024-03-22T05:41:04.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:04 smithi186 ceph-mon[25720]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37 2024-03-22T05:41:06.678 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:06 smithi005 ceph-mon[24793]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 873 B/s rd, 0 op/s 2024-03-22T05:41:06.678 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:06 smithi005 ceph-mon[29523]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 873 B/s rd, 0 op/s 2024-03-22T05:41:06.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:06 smithi186 ceph-mon[25720]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 873 B/s rd, 0 op/s 2024-03-22T05:41:07.368 INFO:teuthology.orchestra.run.smithi005.stdout:true 2024-03-22T05:41:07.939 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:07 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: [22/Mar/2024:05:41:07] ENGINE Bus STOPPING 2024-03-22T05:41:07.939 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:07 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: [22/Mar/2024:05:41:07] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-22T05:41:07.939 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:07 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: [22/Mar/2024:05:41:07] ENGINE Bus STOPPED 2024-03-22T05:41:07.939 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:07 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: [22/Mar/2024:05:41:07] ENGINE Bus STARTING 2024-03-22T05:41:07.940 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:07 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: [22/Mar/2024:05:41:07] ENGINE Serving on http://:::9283 2024-03-22T05:41:07.940 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:07 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: [22/Mar/2024:05:41:07] ENGINE Bus STARTED 2024-03-22T05:41:07.940 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:07 smithi005 ceph-mon[24793]: from='mgr.34976 ' entity='mgr.x' 2024-03-22T05:41:07.940 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:07 smithi005 ceph-mon[24793]: Upgrade: Target is version 19.0.0-2299-gea01217f (squid) 2024-03-22T05:41:07.940 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:07 smithi005 ceph-mon[24793]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, digests ['quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9'] 2024-03-22T05:41:07.940 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:07 smithi005 ceph-mon[24793]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:41:07.940 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:07 smithi005 ceph-mon[24793]: Upgrade: Need to upgrade myself (mgr.x) 2024-03-22T05:41:07.940 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:07 smithi005 ceph-mon[24793]: Upgrade: Need to upgrade myself (mgr.x) 2024-03-22T05:41:07.940 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:07 smithi005 ceph-mon[24793]: Failing over to other MGR 2024-03-22T05:41:07.941 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:07 smithi005 ceph-mon[24793]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-03-22T05:41:07.941 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:07 smithi005 ceph-mon[24793]: from='mgr.34976 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-03-22T05:41:07.941 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:07 smithi005 ceph-mon[24793]: osdmap e104: 8 total, 8 up, 8 in 2024-03-22T05:41:07.941 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:07 smithi005 ceph-mon[24793]: from='client.15399 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:41:07.941 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:07 smithi005 ceph-mon[24793]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 526 B/s rd, 0 op/s 2024-03-22T05:41:07.941 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:07 smithi005 ceph-mon[29523]: from='mgr.34976 ' entity='mgr.x' 2024-03-22T05:41:07.941 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:07 smithi005 ceph-mon[29523]: Upgrade: Target is version 19.0.0-2299-gea01217f (squid) 2024-03-22T05:41:07.941 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:07 smithi005 ceph-mon[29523]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, digests ['quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9'] 2024-03-22T05:41:07.941 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:07 smithi005 ceph-mon[29523]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:41:07.941 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:07 smithi005 ceph-mon[29523]: Upgrade: Need to upgrade myself (mgr.x) 2024-03-22T05:41:07.941 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:07 smithi005 ceph-mon[29523]: Upgrade: Need to upgrade myself (mgr.x) 2024-03-22T05:41:07.941 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:07 smithi005 ceph-mon[29523]: Failing over to other MGR 2024-03-22T05:41:07.941 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:07 smithi005 ceph-mon[29523]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-03-22T05:41:07.941 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:07 smithi005 ceph-mon[29523]: from='mgr.34976 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-03-22T05:41:07.942 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:07 smithi005 ceph-mon[29523]: osdmap e104: 8 total, 8 up, 8 in 2024-03-22T05:41:07.942 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:07 smithi005 ceph-mon[29523]: from='client.15399 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:41:07.942 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:07 smithi005 ceph-mon[29523]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 526 B/s rd, 0 op/s 2024-03-22T05:41:07.973 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:07 smithi186 ceph-mon[25720]: from='mgr.34976 ' entity='mgr.x' 2024-03-22T05:41:07.973 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:07 smithi186 ceph-mon[25720]: Upgrade: Target is version 19.0.0-2299-gea01217f (squid) 2024-03-22T05:41:07.973 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:07 smithi186 ceph-mon[25720]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, digests ['quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9'] 2024-03-22T05:41:07.973 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:07 smithi186 ceph-mon[25720]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:41:07.973 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:07 smithi186 ceph-mon[25720]: Upgrade: Need to upgrade myself (mgr.x) 2024-03-22T05:41:07.973 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:07 smithi186 ceph-mon[25720]: Upgrade: Need to upgrade myself (mgr.x) 2024-03-22T05:41:07.973 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:07 smithi186 ceph-mon[25720]: Failing over to other MGR 2024-03-22T05:41:07.974 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:07 smithi186 ceph-mon[25720]: from='mgr.34976 172.21.15.186:0/2989380924' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-03-22T05:41:07.974 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:07 smithi186 ceph-mon[25720]: from='mgr.34976 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-03-22T05:41:07.974 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:07 smithi186 ceph-mon[25720]: osdmap e104: 8 total, 8 up, 8 in 2024-03-22T05:41:07.974 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:07 smithi186 ceph-mon[25720]: from='client.15399 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:41:07.974 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:07 smithi186 ceph-mon[25720]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 526 B/s rd, 0 op/s 2024-03-22T05:41:07.974 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:41:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: 2024-03-22T05:41:07.693+0000 7efc28d50700 -1 mgr handle_mgr_map I was active but no longer am 2024-03-22T05:41:07.974 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:41:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: ignoring --setuser ceph since I am not root 2024-03-22T05:41:07.974 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:41:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: ignoring --setgroup ceph since I am not root 2024-03-22T05:41:07.974 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:41:07 smithi186 ceph-mgr[45898]: -- 172.21.15.186:0/414983238 <== mon.2 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x558b0f2185a0 con 0x558b0f1ed000 2024-03-22T05:41:08.272 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:41:08 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: 2024-03-22T05:41:08.272+0000 7fdc00ecc200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-22T05:41:08.543 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:41:08 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: 2024-03-22T05:41:08.370+0000 7fdc00ecc200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-22T05:41:08.876 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:08 smithi186 ceph-mon[25720]: from='mgr.34976 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-03-22T05:41:08.877 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:08 smithi186 ceph-mon[25720]: mgrmap e38: y(active, starting, since 0.997971s) 2024-03-22T05:41:08.877 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:08 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-22T05:41:08.877 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:08 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T05:41:08.877 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:08 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T05:41:08.877 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:08 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-22T05:41:08.877 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:08 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T05:41:08.877 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:08 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T05:41:08.877 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:08 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T05:41:08.877 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:08 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T05:41:08.877 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:08 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T05:41:08.877 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:08 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T05:41:08.877 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:08 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-22T05:41:08.877 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:08 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-22T05:41:08.877 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:08 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-22T05:41:08.878 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:08 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-22T05:41:08.878 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:08 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-22T05:41:08.878 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:08 smithi186 ceph-mon[25720]: Manager daemon y is now available 2024-03-22T05:41:08.878 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:08 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:41:08.878 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:08 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:41:08.878 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:08 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-22T05:41:08.878 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:08 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-22T05:41:08.878 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:08 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-22T05:41:08.878 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:08 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-22T05:41:08.878 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:08 smithi186 ceph-mon[25720]: [22/Mar/2024:05:41:08] ENGINE Bus STARTING 2024-03-22T05:41:08.878 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:08 smithi186 ceph-mon[25720]: [22/Mar/2024:05:41:08] ENGINE Serving on https://172.21.15.5:7150 2024-03-22T05:41:08.878 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:08 smithi186 ceph-mon[25720]: [22/Mar/2024:05:41:08] ENGINE Serving on http://172.21.15.5:8765 2024-03-22T05:41:08.878 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:08 smithi186 ceph-mon[25720]: [22/Mar/2024:05:41:08] ENGINE Bus STARTED 2024-03-22T05:41:08.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:08 smithi005 ceph-mon[24793]: from='mgr.34976 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-03-22T05:41:08.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:08 smithi005 ceph-mon[24793]: mgrmap e38: y(active, starting, since 0.997971s) 2024-03-22T05:41:08.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:08 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-22T05:41:08.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:08 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T05:41:08.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:08 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T05:41:08.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:08 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-22T05:41:08.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:08 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T05:41:08.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:08 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T05:41:08.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:08 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T05:41:08.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:08 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T05:41:08.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:08 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T05:41:08.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:08 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T05:41:08.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:08 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-22T05:41:08.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:08 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-22T05:41:08.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:08 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-22T05:41:08.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:08 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-22T05:41:08.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:08 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-22T05:41:08.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:08 smithi005 ceph-mon[24793]: Manager daemon y is now available 2024-03-22T05:41:08.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:08 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:41:08.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:08 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:41:08.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:08 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-22T05:41:08.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:08 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-22T05:41:08.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:08 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-22T05:41:08.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:08 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-22T05:41:08.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:08 smithi005 ceph-mon[24793]: [22/Mar/2024:05:41:08] ENGINE Bus STARTING 2024-03-22T05:41:08.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:08 smithi005 ceph-mon[24793]: [22/Mar/2024:05:41:08] ENGINE Serving on https://172.21.15.5:7150 2024-03-22T05:41:08.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:08 smithi005 ceph-mon[24793]: [22/Mar/2024:05:41:08] ENGINE Serving on http://172.21.15.5:8765 2024-03-22T05:41:08.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:08 smithi005 ceph-mon[24793]: [22/Mar/2024:05:41:08] ENGINE Bus STARTED 2024-03-22T05:41:08.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:08 smithi005 ceph-mon[29523]: from='mgr.34976 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-03-22T05:41:08.955 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:08 smithi005 ceph-mon[29523]: mgrmap e38: y(active, starting, since 0.997971s) 2024-03-22T05:41:08.955 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:08 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-22T05:41:08.955 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:08 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T05:41:08.955 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:08 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T05:41:08.955 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:08 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-22T05:41:08.955 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:08 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T05:41:08.955 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:08 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T05:41:08.955 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:08 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T05:41:08.955 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:08 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T05:41:08.955 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:08 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T05:41:08.955 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:08 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T05:41:08.955 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:08 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-22T05:41:08.955 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:08 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-22T05:41:08.955 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:08 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-22T05:41:08.955 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:08 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-22T05:41:08.956 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:08 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-22T05:41:08.956 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:08 smithi005 ceph-mon[29523]: Manager daemon y is now available 2024-03-22T05:41:08.956 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:08 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:41:08.956 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:08 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:41:08.956 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:08 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-22T05:41:08.956 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:08 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-22T05:41:08.956 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:08 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-22T05:41:08.956 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:08 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-22T05:41:08.956 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:08 smithi005 ceph-mon[29523]: [22/Mar/2024:05:41:08] ENGINE Bus STARTING 2024-03-22T05:41:08.956 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:08 smithi005 ceph-mon[29523]: [22/Mar/2024:05:41:08] ENGINE Serving on https://172.21.15.5:7150 2024-03-22T05:41:08.956 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:08 smithi005 ceph-mon[29523]: [22/Mar/2024:05:41:08] ENGINE Serving on http://172.21.15.5:8765 2024-03-22T05:41:08.956 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:08 smithi005 ceph-mon[29523]: [22/Mar/2024:05:41:08] ENGINE Bus STARTED 2024-03-22T05:41:09.052 INFO:teuthology.orchestra.run.smithi005.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T05:41:09.052 INFO:teuthology.orchestra.run.smithi005.stdout:alertmanager.a smithi005 *:9093,9094 running (12m) 3m ago 20m 22.8M - 0.23.0 ba2b418f427c fd7dc5dbe3e5 2024-03-22T05:41:09.052 INFO:teuthology.orchestra.run.smithi005.stdout:grafana.a smithi186 *:3000 error 3m ago 20m - - 2024-03-22T05:41:09.052 INFO:teuthology.orchestra.run.smithi005.stdout:iscsi.foo.smithi005.mcdwbp smithi005 error 3m ago 19m 55.0M - 3.5 e1d6a67b021e 5215e03bf81b 2024-03-22T05:41:09.052 INFO:teuthology.orchestra.run.smithi005.stdout:mgr.x smithi186 *:8443,9283 running (15m) 3m ago 25m 494M - 19.0.0-2299-gea01217f aa461e7c5e7f 66924e151e7b 2024-03-22T05:41:09.052 INFO:teuthology.orchestra.run.smithi005.stdout:mgr.y smithi005 *:8443,9283,8765 running (9m) 3m ago 26m 72.0M - 19.0.0-2299-gea01217f aa461e7c5e7f 62a813eebc0d 2024-03-22T05:41:09.052 INFO:teuthology.orchestra.run.smithi005.stdout:mon.a smithi005 running (26m) 3m ago 26m 72.2M 2048M 17.2.0 e1d6a67b021e 38e0bba4bb03 2024-03-22T05:41:09.052 INFO:teuthology.orchestra.run.smithi005.stdout:mon.b smithi186 running (25m) 3m ago 25m 49.0M 2048M 17.2.0 e1d6a67b021e 323f53f932f6 2024-03-22T05:41:09.052 INFO:teuthology.orchestra.run.smithi005.stdout:mon.c smithi005 running (25m) 3m ago 25m 53.3M 2048M 17.2.0 e1d6a67b021e 00ec24d38612 2024-03-22T05:41:09.052 INFO:teuthology.orchestra.run.smithi005.stdout:node-exporter.a smithi005 *:9100 running (12m) 3m ago 21m 16.2M - 1.3.1 1dbe0e931976 a9085d6042eb 2024-03-22T05:41:09.052 INFO:teuthology.orchestra.run.smithi005.stdout:node-exporter.b smithi186 *:9100 running (12m) 3m ago 21m 18.4M - 1.3.1 1dbe0e931976 74aa1181ff71 2024-03-22T05:41:09.052 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0 smithi005 running (24m) 3m ago 24m 52.6M 2993M 17.2.0 e1d6a67b021e ef5c7053dd37 2024-03-22T05:41:09.052 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1 smithi005 running (24m) 3m ago 24m 55.1M 2993M 17.2.0 e1d6a67b021e a1e7ee1017b6 2024-03-22T05:41:09.053 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2 smithi005 running (23m) 3m ago 23m 48.1M 2993M 17.2.0 e1d6a67b021e 462ea63ad0e5 2024-03-22T05:41:09.053 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3 smithi005 running (23m) 3m ago 23m 55.1M 2993M 17.2.0 e1d6a67b021e b25432a7239b 2024-03-22T05:41:09.053 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4 smithi186 running (23m) 3m ago 23m 54.0M 3248M 17.2.0 e1d6a67b021e cd963671f340 2024-03-22T05:41:09.053 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5 smithi186 running (22m) 3m ago 22m 50.8M 3248M 17.2.0 e1d6a67b021e f98c87f1a2cd 2024-03-22T05:41:09.053 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6 smithi186 running (22m) 3m ago 22m 50.1M 3248M 17.2.0 e1d6a67b021e 704ac49c68c4 2024-03-22T05:41:09.053 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7 smithi186 running (22m) 3m ago 22m 52.8M 3248M 17.2.0 e1d6a67b021e f415b90edf63 2024-03-22T05:41:09.053 INFO:teuthology.orchestra.run.smithi005.stdout:prometheus.a smithi186 *:9095 running (3m) 3m ago 21m 42.1M - 2.33.4 514e6a882f6e bf952ea306b4 2024-03-22T05:41:09.053 INFO:teuthology.orchestra.run.smithi005.stdout:rgw.foo.smithi005.exdhqu smithi005 *:8000 running (19m) 3m ago 19m 77.5M - 17.2.0 e1d6a67b021e 6bb27fe11553 2024-03-22T05:41:09.053 INFO:teuthology.orchestra.run.smithi005.stdout:rgw.foo.smithi186.mcgodk smithi186 *:8000 running (19m) 3m ago 19m 75.7M - 17.2.0 e1d6a67b021e 19b5c20a34f7 2024-03-22T05:41:09.164 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:41:09 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: 2024-03-22T05:41:09.064+0000 7fdc00ecc200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-22T05:41:09.419 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:41:09 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: 2024-03-22T05:41:09.164+0000 7fdc00ecc200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-22T05:41:09.419 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:41:09 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: 2024-03-22T05:41:09.309+0000 7fdc00ecc200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-22T05:41:09.468 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-03-22T05:41:09.468 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": { 2024-03-22T05:41:09.468 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-03-22T05:41:09.468 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:41:09.468 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr": { 2024-03-22T05:41:09.468 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev)": 1 2024-03-22T05:41:09.468 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:41:09.468 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": { 2024-03-22T05:41:09.468 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-22T05:41:09.469 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:41:09.469 INFO:teuthology.orchestra.run.smithi005.stdout: "mds": {}, 2024-03-22T05:41:09.469 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw": { 2024-03-22T05:41:09.469 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-22T05:41:09.469 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:41:09.469 INFO:teuthology.orchestra.run.smithi005.stdout: "overall": { 2024-03-22T05:41:09.469 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-03-22T05:41:09.469 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev)": 1 2024-03-22T05:41:09.469 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-03-22T05:41:09.469 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-03-22T05:41:09.704 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:41:09 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: 2024-03-22T05:41:09.418+0000 7fdc00ecc200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-22T05:41:09.704 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:41:09 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: 2024-03-22T05:41:09.502+0000 7fdc00ecc200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-22T05:41:09.733 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:09 smithi005 ceph-mon[29523]: mgrmap e39: y(active, since 2s) 2024-03-22T05:41:09.733 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:09 smithi005 ceph-mon[29523]: from='client.25315 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:41:09.733 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:09 smithi005 ceph-mon[29523]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:41:09.733 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:09 smithi005 ceph-mon[29523]: from='client.25339 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:41:09.733 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:09 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/370244395' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:41:09.734 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:09 smithi005 ceph-mon[24793]: mgrmap e39: y(active, since 2s) 2024-03-22T05:41:09.734 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:09 smithi005 ceph-mon[24793]: from='client.25315 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:41:09.734 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:09 smithi005 ceph-mon[24793]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:41:09.734 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:09 smithi005 ceph-mon[24793]: from='client.25339 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:41:09.734 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:09 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/370244395' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:41:09.830 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-03-22T05:41:09.830 INFO:teuthology.orchestra.run.smithi005.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9", 2024-03-22T05:41:09.830 INFO:teuthology.orchestra.run.smithi005.stdout: "in_progress": true, 2024-03-22T05:41:09.830 INFO:teuthology.orchestra.run.smithi005.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-03-22T05:41:09.830 INFO:teuthology.orchestra.run.smithi005.stdout: "services_complete": [ 2024-03-22T05:41:09.831 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr" 2024-03-22T05:41:09.831 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-03-22T05:41:09.831 INFO:teuthology.orchestra.run.smithi005.stdout: "progress": "2/2 daemons upgraded", 2024-03-22T05:41:09.831 INFO:teuthology.orchestra.run.smithi005.stdout: "message": "", 2024-03-22T05:41:09.831 INFO:teuthology.orchestra.run.smithi005.stdout: "is_paused": false 2024-03-22T05:41:09.831 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-03-22T05:41:09.984 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:41:09 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: 2024-03-22T05:41:09.780+0000 7fdc00ecc200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-22T05:41:09.984 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:41:09 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: 2024-03-22T05:41:09.896+0000 7fdc00ecc200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-22T05:41:09.984 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:09 smithi186 ceph-mon[25720]: mgrmap e39: y(active, since 2s) 2024-03-22T05:41:09.984 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:09 smithi186 ceph-mon[25720]: from='client.25315 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:41:09.984 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:09 smithi186 ceph-mon[25720]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:41:09.985 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:09 smithi186 ceph-mon[25720]: from='client.25339 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:41:09.985 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:09 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/370244395' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:41:10.289 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:41:09 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: 2024-03-22T05:41:09.984+0000 7fdc00ecc200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-22T05:41:10.290 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:41:10 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: 2024-03-22T05:41:10.090+0000 7fdc00ecc200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-22T05:41:10.290 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:41:10 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: 2024-03-22T05:41:10.289+0000 7fdc00ecc200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-22T05:41:10.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:10 smithi005 ceph-mon[29523]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:41:10.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:10 smithi005 ceph-mon[29523]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 2 failed cephadm daemon(s)) 2024-03-22T05:41:10.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:10 smithi005 ceph-mon[29523]: Cluster is now healthy 2024-03-22T05:41:10.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:10 smithi005 ceph-mon[29523]: mgrmap e40: y(active, since 3s) 2024-03-22T05:41:10.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:10 smithi005 ceph-mon[29523]: from='client.25351 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:41:10.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:10 smithi005 ceph-mon[24793]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:41:10.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:10 smithi005 ceph-mon[24793]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 2 failed cephadm daemon(s)) 2024-03-22T05:41:10.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:10 smithi005 ceph-mon[24793]: Cluster is now healthy 2024-03-22T05:41:10.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:10 smithi005 ceph-mon[24793]: mgrmap e40: y(active, since 3s) 2024-03-22T05:41:10.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:10 smithi005 ceph-mon[24793]: from='client.25351 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:41:11.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:10 smithi186 ceph-mon[25720]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:41:11.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:10 smithi186 ceph-mon[25720]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 2 failed cephadm daemon(s)) 2024-03-22T05:41:11.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:10 smithi186 ceph-mon[25720]: Cluster is now healthy 2024-03-22T05:41:11.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:10 smithi186 ceph-mon[25720]: mgrmap e40: y(active, since 3s) 2024-03-22T05:41:11.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:10 smithi186 ceph-mon[25720]: from='client.25351 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:41:11.543 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:41:11 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: 2024-03-22T05:41:11.287+0000 7fdc00ecc200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-22T05:41:11.543 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:41:11 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: 2024-03-22T05:41:11.419+0000 7fdc00ecc200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-22T05:41:11.901 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:11 smithi005 ceph-mon[24793]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:41:11.901 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:11 smithi005 ceph-mon[29523]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:41:12.043 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:41:11 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: 2024-03-22T05:41:11.767+0000 7fdc00ecc200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-22T05:41:12.043 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:41:11 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: 2024-03-22T05:41:11.875+0000 7fdc00ecc200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-22T05:41:12.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:11 smithi186 ceph-mon[25720]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:41:12.543 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:41:12 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: 2024-03-22T05:41:12.091+0000 7fdc00ecc200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-22T05:41:12.543 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:41:12 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: 2024-03-22T05:41:12.289+0000 7fdc00ecc200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-22T05:41:12.875 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:41:12 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: 2024-03-22T05:41:12.596+0000 7fdc00ecc200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-22T05:41:12.875 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:12 smithi186 ceph-mon[25720]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:41:12.875 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:12 smithi186 ceph-mon[25720]: mgrmap e41: y(active, since 5s) 2024-03-22T05:41:12.875 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:12 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:12.875 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:12 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:12.875 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:12 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:12.876 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:12 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:13.137 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:41:12 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: 2024-03-22T05:41:12.875+0000 7fdc00ecc200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-22T05:41:13.138 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:41:13 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: 2024-03-22T05:41:13.137+0000 7fdc00ecc200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-22T05:41:13.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:12 smithi005 ceph-mon[29523]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:41:13.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:12 smithi005 ceph-mon[29523]: mgrmap e41: y(active, since 5s) 2024-03-22T05:41:13.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:12 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:13.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:12 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:13.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:12 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:13.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:12 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:13.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:12 smithi005 ceph-mon[24793]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:41:13.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:12 smithi005 ceph-mon[24793]: mgrmap e41: y(active, since 5s) 2024-03-22T05:41:13.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:12 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:13.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:12 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:13.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:12 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:13.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:12 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:14.293 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:41:13 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: 2024-03-22T05:41:13.997+0000 7fdc00ecc200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-22T05:41:14.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:14 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:14.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:14 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:14.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:14 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:14.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:14 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:14.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:14 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:41:14.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:14 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:41:14.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:14 smithi186 ceph-mon[25720]: Updating smithi005:/etc/ceph/ceph.conf 2024-03-22T05:41:14.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:14 smithi186 ceph-mon[25720]: Updating smithi186:/etc/ceph/ceph.conf 2024-03-22T05:41:14.452 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:41:14 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[59009]: level=warn ts=2024-03-22T05:41:14.246Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi186.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.186:8443: connect: connection refused" 2024-03-22T05:41:14.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:14 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:14.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:14 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:14.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:14 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:14.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:14 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:14.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:14 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:41:14.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:14 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:41:14.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:14 smithi005 ceph-mon[24793]: Updating smithi005:/etc/ceph/ceph.conf 2024-03-22T05:41:14.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:14 smithi005 ceph-mon[24793]: Updating smithi186:/etc/ceph/ceph.conf 2024-03-22T05:41:14.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:14 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:14.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:14 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:14.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:14 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:14.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:14 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:14.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:14 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:41:14.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:14 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:41:14.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:14 smithi005 ceph-mon[29523]: Updating smithi005:/etc/ceph/ceph.conf 2024-03-22T05:41:14.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:14 smithi005 ceph-mon[29523]: Updating smithi186:/etc/ceph/ceph.conf 2024-03-22T05:41:14.931 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:41:14 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: 2024-03-22T05:41:14.604+0000 7fdc00ecc200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-22T05:41:14.931 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:41:14 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: 2024-03-22T05:41:14.707+0000 7fdc00ecc200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-22T05:41:14.931 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:41:14 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: 2024-03-22T05:41:14.931+0000 7fdc00ecc200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-22T05:41:15.293 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:41:14 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: [22/Mar/2024:05:41:14] ENGINE Bus STARTING 2024-03-22T05:41:15.294 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:41:15 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: CherryPy Checker: 2024-03-22T05:41:15.294 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:41:15 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: The Application mounted at '' has an empty config. 2024-03-22T05:41:15.294 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:41:15 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: 2024-03-22T05:41:15.294 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:41:15 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: [22/Mar/2024:05:41:15] ENGINE Serving on http://:::9283 2024-03-22T05:41:15.294 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:41:15 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[45894]: [22/Mar/2024:05:41:15] ENGINE Bus STARTED 2024-03-22T05:41:15.434 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:41:15 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[59009]: level=info ts=2024-03-22T05:41:15.112Z caller=notify.go:735 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify success" attempts=3 2024-03-22T05:41:15.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:15 smithi005 ceph-mon[24793]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:41:15.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:15 smithi005 ceph-mon[24793]: Updating smithi005:/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T05:41:15.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:15 smithi005 ceph-mon[24793]: Updating smithi186:/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T05:41:15.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:15 smithi005 ceph-mon[24793]: Updating smithi005:/etc/ceph/ceph.client.admin.keyring 2024-03-22T05:41:15.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:15 smithi005 ceph-mon[24793]: Updating smithi186:/etc/ceph/ceph.client.admin.keyring 2024-03-22T05:41:15.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:15 smithi005 ceph-mon[24793]: Updating smithi005:/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-22T05:41:15.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:15 smithi005 ceph-mon[24793]: Updating smithi186:/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-22T05:41:15.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:15 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:15.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:15 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:15.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:15 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:15.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:15 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:15.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:15 smithi005 ceph-mon[24793]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 20 op/s 2024-03-22T05:41:15.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:15 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:15.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:15 smithi005 ceph-mon[24793]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:41:15.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:15 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:41:15.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:15 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:41:15.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:15 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:41:15.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:15 smithi005 ceph-mon[24793]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:41:15.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:15 smithi005 ceph-mon[24793]: Standby manager daemon x started 2024-03-22T05:41:15.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:15 smithi005 ceph-mon[24793]: from='mgr.? 172.21.15.186:0/3074464232' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-22T05:41:15.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:15 smithi005 ceph-mon[24793]: from='mgr.? 172.21.15.186:0/3074464232' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-22T05:41:15.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:15 smithi005 ceph-mon[24793]: from='mgr.? 172.21.15.186:0/3074464232' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-22T05:41:15.705 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:15 smithi005 ceph-mon[24793]: from='mgr.? 172.21.15.186:0/3074464232' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-22T05:41:15.705 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:15 smithi005 ceph-mon[29523]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:41:15.705 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:15 smithi005 ceph-mon[29523]: Updating smithi005:/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T05:41:15.705 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:15 smithi005 ceph-mon[29523]: Updating smithi186:/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T05:41:15.705 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:15 smithi005 ceph-mon[29523]: Updating smithi005:/etc/ceph/ceph.client.admin.keyring 2024-03-22T05:41:15.705 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:15 smithi005 ceph-mon[29523]: Updating smithi186:/etc/ceph/ceph.client.admin.keyring 2024-03-22T05:41:15.705 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:15 smithi005 ceph-mon[29523]: Updating smithi005:/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-22T05:41:15.705 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:15 smithi005 ceph-mon[29523]: Updating smithi186:/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-22T05:41:15.705 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:15 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:15.705 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:15 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:15.705 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:15 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:15.705 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:15 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:15.706 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:15 smithi005 ceph-mon[29523]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 20 op/s 2024-03-22T05:41:15.706 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:15 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:15.706 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:15 smithi005 ceph-mon[29523]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:41:15.706 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:15 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:41:15.706 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:15 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:41:15.706 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:15 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:41:15.706 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:15 smithi005 ceph-mon[29523]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:41:15.706 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:15 smithi005 ceph-mon[29523]: Standby manager daemon x started 2024-03-22T05:41:15.706 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:15 smithi005 ceph-mon[29523]: from='mgr.? 172.21.15.186:0/3074464232' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-22T05:41:15.706 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:15 smithi005 ceph-mon[29523]: from='mgr.? 172.21.15.186:0/3074464232' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-22T05:41:15.706 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:15 smithi005 ceph-mon[29523]: from='mgr.? 172.21.15.186:0/3074464232' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-22T05:41:15.706 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:15 smithi005 ceph-mon[29523]: from='mgr.? 172.21.15.186:0/3074464232' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-22T05:41:16.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:15 smithi186 ceph-mon[25720]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:41:16.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:15 smithi186 ceph-mon[25720]: Updating smithi005:/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T05:41:16.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:15 smithi186 ceph-mon[25720]: Updating smithi186:/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T05:41:16.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:15 smithi186 ceph-mon[25720]: Updating smithi005:/etc/ceph/ceph.client.admin.keyring 2024-03-22T05:41:16.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:15 smithi186 ceph-mon[25720]: Updating smithi186:/etc/ceph/ceph.client.admin.keyring 2024-03-22T05:41:16.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:15 smithi186 ceph-mon[25720]: Updating smithi005:/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-22T05:41:16.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:15 smithi186 ceph-mon[25720]: Updating smithi186:/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-22T05:41:16.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:15 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:16.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:15 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:16.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:15 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:16.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:15 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:16.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:15 smithi186 ceph-mon[25720]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 20 op/s 2024-03-22T05:41:16.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:15 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:16.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:15 smithi186 ceph-mon[25720]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:41:16.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:15 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:41:16.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:15 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:41:16.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:15 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:41:16.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:15 smithi186 ceph-mon[25720]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:41:16.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:15 smithi186 ceph-mon[25720]: Standby manager daemon x started 2024-03-22T05:41:16.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:15 smithi186 ceph-mon[25720]: from='mgr.? 172.21.15.186:0/3074464232' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-22T05:41:16.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:15 smithi186 ceph-mon[25720]: from='mgr.? 172.21.15.186:0/3074464232' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-22T05:41:16.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:15 smithi186 ceph-mon[25720]: from='mgr.? 172.21.15.186:0/3074464232' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-22T05:41:16.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:15 smithi186 ceph-mon[25720]: from='mgr.? 172.21.15.186:0/3074464232' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-22T05:41:16.562 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:16 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:41:16.451+0000 7f487c563700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:41:16.562 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:16 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:41:16.562 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:16 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:41:16.562 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:16 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:41:16.562 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:16 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:41:16.563 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:16 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:41:16.563 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:16 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:41:16.563 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:16 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:41:16.563 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:16 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:41:16.563 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:16 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:41:16.563 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:16 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:41:16.563 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:16 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:41:16.563 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:16 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:41:16.563 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:16 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:41:16.563 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:16 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:41:16.563 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:16 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:41:16.563 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:16 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:41:16.563 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:16 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:41:16.563 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:16 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:41:16.564 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:16 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:41:16.564 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:16 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:41:16.564 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:16 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:41:16.564 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:16 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:41:16.564 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:16 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:41:16.564 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:16 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:41:16.564 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:16 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:41:16.564 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:16 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:41:16.564 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:16 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:41:16.565 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:16 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:41:16.565 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:16 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return future.result(timeout) 2024-03-22T05:41:16.565 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:16 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:41:16.565 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:16 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.__get_result() 2024-03-22T05:41:16.565 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:16 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:41:16.565 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:16 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: raise self._exception 2024-03-22T05:41:16.565 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:16 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:41:16.565 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:16 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: config_blobs=daemon_spec.final_config, 2024-03-22T05:41:16.565 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:16 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:41:16.565 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:16 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:41:16.565 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:16 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:41:16.565 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:16 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:41:16.565 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:16 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:41:16.565 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:16 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:41:16.566 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:16 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:41:16.566 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:16 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:41:16.566 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:16 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:41:16.566 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:16 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:41:16.566 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:16 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:41:16.566 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:16 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:41:16.566 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:16 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:41:16.566 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:16 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:41:16.566 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:16 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:41:16.566 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:16 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:41:16.566 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:16 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:41:16.566 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:16 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:41:16.566 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:16 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:41:16.566 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:16 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:41:16.566 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:16 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:41:16.567 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:16 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:41:16.567 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:16 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:41:16.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:16 smithi005 ceph-mon[24793]: Health check failed: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-22T05:41:16.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:16 smithi005 ceph-mon[24793]: mgrmap e42: y(active, since 8s), standbys: x 2024-03-22T05:41:16.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:16 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-22T05:41:16.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:16 smithi005 ceph-mon[29523]: Health check failed: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-22T05:41:16.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:16 smithi005 ceph-mon[29523]: mgrmap e42: y(active, since 8s), standbys: x 2024-03-22T05:41:16.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:16 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-22T05:41:17.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:16 smithi186 ceph-mon[25720]: Health check failed: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-22T05:41:17.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:16 smithi186 ceph-mon[25720]: mgrmap e42: y(active, since 8s), standbys: x 2024-03-22T05:41:17.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:16 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-22T05:41:17.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:17 smithi005 ceph-mon[24793]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:41:17.952 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:41:17.952 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:41:17.952 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:41:17.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:41:17.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:41:17.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:41:17.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:41:17.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:41:17.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:41:17.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:41:17.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:41:17.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:41:17.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:41:17.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:41:17.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:41:17.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:41:17.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:41:17.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:41:17.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:41:17.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:41:17.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:41:17.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:41:17.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:41:17.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:41:17.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:41:17.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:41:17.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:41:17.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:41:17.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: return future.result(timeout) 2024-03-22T05:41:17.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:41:17.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.__get_result() 2024-03-22T05:41:17.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:41:17.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: raise self._exception 2024-03-22T05:41:17.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:41:17.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:41:17.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:41:17.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:41:17.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:41:17.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:41:17.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:41:17.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:41:17.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:41:17.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:41:17.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:41:17.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:41:17.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:41:17.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:41:17.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:41:17.956 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:41:17.956 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:41:17.956 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:41:17.956 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:41:17.956 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:41:17.956 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:41:17.956 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:41:17.956 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:41:17.956 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:41:17.956 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:41:17.956 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:17 smithi005 ceph-mon[24793]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-22T05:41:17.956 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:17 smithi005 ceph-mon[24793]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 15 op/s 2024-03-22T05:41:17.957 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:17 smithi005 ceph-mon[29523]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:41:17.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:41:17.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:41:17.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:41:17.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:41:17.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:41:17.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:41:17.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:41:17.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:41:17.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:41:17.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:41:17.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:41:17.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:41:17.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:41:17.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:41:17.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:41:17.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:41:17.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:41:17.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:41:17.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:41:17.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:41:17.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:41:17.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:41:17.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:41:17.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:41:17.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:41:17.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:41:17.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:41:17.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:41:17.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: return future.result(timeout) 2024-03-22T05:41:17.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:41:17.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.__get_result() 2024-03-22T05:41:17.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:41:17.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: raise self._exception 2024-03-22T05:41:17.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:41:17.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:41:17.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:41:17.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:41:17.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:41:17.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:41:17.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:41:17.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:41:17.960 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:41:17.960 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:41:17.960 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:41:17.960 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:41:17.960 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:41:17.960 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:41:17.960 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:41:17.960 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:41:17.960 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:41:17.960 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:41:17.960 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:41:17.960 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:41:17.960 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:41:17.960 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:41:17.961 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:41:17.961 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:41:17.961 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:41:17.961 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:17 smithi005 ceph-mon[29523]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-22T05:41:17.961 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:17 smithi005 ceph-mon[29523]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 15 op/s 2024-03-22T05:41:18.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:17 smithi186 ceph-mon[25720]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:41:18.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:41:18.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:41:18.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:41:18.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:41:18.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:41:18.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:41:18.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:41:18.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:41:18.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:41:18.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:41:18.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:41:18.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:41:18.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:41:18.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:41:18.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:41:18.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:41:18.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:41:18.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:41:18.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:41:18.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:41:18.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:41:18.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:41:18.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:41:18.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:41:18.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:41:18.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:41:18.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:41:18.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:41:18.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: return future.result(timeout) 2024-03-22T05:41:18.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:41:18.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.__get_result() 2024-03-22T05:41:18.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:41:18.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: raise self._exception 2024-03-22T05:41:18.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:41:18.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:41:18.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:41:18.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:41:18.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:41:18.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:41:18.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:41:18.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:41:18.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:41:18.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:41:18.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:41:18.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:41:18.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:41:18.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:41:18.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:41:18.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:41:18.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:41:18.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:41:18.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:41:18.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:41:18.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:41:18.048 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:41:18.048 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:41:18.048 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:41:18.048 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:41:18.048 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:17 smithi186 ceph-mon[25720]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-22T05:41:18.048 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:17 smithi186 ceph-mon[25720]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 15 op/s 2024-03-22T05:41:18.731 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:18 smithi186 ceph-mon[25720]: Reconfiguring daemon prometheus.a on smithi186 2024-03-22T05:41:18.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:18 smithi005 ceph-mon[24793]: Reconfiguring daemon prometheus.a on smithi186 2024-03-22T05:41:18.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:18 smithi005 ceph-mon[29523]: Reconfiguring daemon prometheus.a on smithi186 2024-03-22T05:41:19.721 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:19 smithi186 ceph-mon[25720]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-22T05:41:19.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:19 smithi005 ceph-mon[24793]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-22T05:41:19.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:19 smithi005 ceph-mon[29523]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-22T05:41:20.044 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:41:19 smithi186 systemd[1]: Stopping Ceph prometheus.a for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:41:20.044 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:41:19 smithi186 bash[60685]: Error: no container with name or ID "ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus.a" found: no such container 2024-03-22T05:41:20.044 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:41:19 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[57980]: ts=2024-03-22T05:41:19.900Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-22T05:41:20.044 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:41:19 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[57980]: ts=2024-03-22T05:41:19.901Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-03-22T05:41:20.044 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:41:19 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[57980]: ts=2024-03-22T05:41:19.901Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-03-22T05:41:20.044 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:41:19 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[57980]: ts=2024-03-22T05:41:19.901Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-03-22T05:41:20.044 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:41:19 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[57980]: ts=2024-03-22T05:41:19.901Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-03-22T05:41:20.044 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:41:19 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[57980]: ts=2024-03-22T05:41:19.901Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-03-22T05:41:20.044 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:41:19 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[57980]: ts=2024-03-22T05:41:19.901Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-22T05:41:20.044 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:41:19 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[57980]: ts=2024-03-22T05:41:19.901Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-03-22T05:41:20.044 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:41:19 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[57980]: ts=2024-03-22T05:41:19.901Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-03-22T05:41:20.045 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:41:19 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[57980]: ts=2024-03-22T05:41:19.901Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-03-22T05:41:20.045 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:41:19 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[57980]: ts=2024-03-22T05:41:19.901Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-03-22T05:41:20.045 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:41:19 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[57980]: ts=2024-03-22T05:41:19.901Z caller=main.go:1066 level=info msg="See you next time!" 2024-03-22T05:41:20.045 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:41:19 smithi186 podman[60698]: 2024-03-22 05:41:19.911126396 +0000 UTC m=+0.106442244 container died bf952ea306b42abec9d95c50ed050f5e7d5e18c2315fc833f132f42bd3816a74 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-22T05:41:20.475 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:41:20 smithi186 podman[60698]: 2024-03-22 05:41:20.250405354 +0000 UTC m=+0.445721204 container cleanup bf952ea306b42abec9d95c50ed050f5e7d5e18c2315fc833f132f42bd3816a74 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-22T05:41:20.475 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:41:20 smithi186 bash[60698]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a 2024-03-22T05:41:20.735 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:41:20 smithi186 podman[60713]: 2024-03-22 05:41:20.476577529 +0000 UTC m=+0.568638706 container remove bf952ea306b42abec9d95c50ed050f5e7d5e18c2315fc833f132f42bd3816a74 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-22T05:41:20.735 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:41:20 smithi186 bash[60726]: Error: no container with name or ID "ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus.a" found: no such container 2024-03-22T05:41:20.735 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:41:20 smithi186 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-03-22T05:41:20.735 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:41:20 smithi186 systemd[1]: Stopped Ceph prometheus.a for f59f36e2-e80a-11ee-95cd-87774f69a715. 2024-03-22T05:41:20.735 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:41:20 smithi186 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@prometheus.a.service: Consumed 1.422s CPU time. 2024-03-22T05:41:20.735 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:41:20 smithi186 systemd[1]: Starting Ceph prometheus.a for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:41:21.043 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:41:20 smithi186 podman[60793]: 2024-03-22 05:41:20.639929501 +0000 UTC m=+0.018687888 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-03-22T05:41:21.043 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:41:20 smithi186 podman[60793]: 2024-03-22 05:41:20.78047428 +0000 UTC m=+0.159232648 container create 8596d431e5578a06c9264837e812c33e534a2dc1b1d0597ca291be9ba361cc05 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-22T05:41:21.504 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:41:21 smithi186 podman[60793]: 2024-03-22 05:41:21.046757995 +0000 UTC m=+0.425516373 container init 8596d431e5578a06c9264837e812c33e534a2dc1b1d0597ca291be9ba361cc05 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-22T05:41:21.505 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:41:21 smithi186 podman[60793]: 2024-03-22 05:41:21.049285573 +0000 UTC m=+0.428043942 container start 8596d431e5578a06c9264837e812c33e534a2dc1b1d0597ca291be9ba361cc05 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-22T05:41:21.505 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:41:21 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[60807]: ts=2024-03-22T05:41:21.072Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-03-22T05:41:21.505 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:41:21 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[60807]: ts=2024-03-22T05:41:21.072Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-03-22T05:41:21.505 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:41:21 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[60807]: ts=2024-03-22T05:41:21.072Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-03-22T05:41:21.505 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:41:21 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[60807]: ts=2024-03-22T05:41:21.072Z caller=main.go:518 level=info host_details="(Linux 5.14.0-430.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024 x86_64 smithi186 (none))" 2024-03-22T05:41:21.505 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:41:21 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[60807]: ts=2024-03-22T05:41:21.072Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-22T05:41:21.505 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:41:21 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[60807]: ts=2024-03-22T05:41:21.072Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-22T05:41:21.505 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:41:21 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[60807]: ts=2024-03-22T05:41:21.074Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-03-22T05:41:21.505 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:41:21 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[60807]: ts=2024-03-22T05:41:21.074Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-03-22T05:41:21.505 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:41:21 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[60807]: ts=2024-03-22T05:41:21.075Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-03-22T05:41:21.505 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:41:21 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[60807]: ts=2024-03-22T05:41:21.077Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-22T05:41:21.505 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:41:21 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[60807]: ts=2024-03-22T05:41:21.077Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.623µs 2024-03-22T05:41:21.505 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:41:21 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[60807]: ts=2024-03-22T05:41:21.077Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-22T05:41:21.506 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:41:21 smithi186 bash[60793]: 8596d431e5578a06c9264837e812c33e534a2dc1b1d0597ca291be9ba361cc05 2024-03-22T05:41:21.506 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:41:21 smithi186 systemd[1]: Started Ceph prometheus.a for f59f36e2-e80a-11ee-95cd-87774f69a715. 2024-03-22T05:41:21.506 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:41:21 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[60807]: ts=2024-03-22T05:41:21.282Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=5 2024-03-22T05:41:21.506 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:41:21 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[60807]: ts=2024-03-22T05:41:21.505Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=5 2024-03-22T05:41:21.793 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:41:21 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[60807]: ts=2024-03-22T05:41:21.740Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=5 2024-03-22T05:41:21.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:21 smithi186 ceph-mon[25720]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 77 op/s 2024-03-22T05:41:21.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:21 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:21.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:21 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:21.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:21 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-22T05:41:21.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:21 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:41:21.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:21 smithi005 ceph-mon[24793]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 77 op/s 2024-03-22T05:41:21.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:21 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:21.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:21 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:21.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:21 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-22T05:41:21.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:21 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:41:21.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:21 smithi005 ceph-mon[29523]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 77 op/s 2024-03-22T05:41:21.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:21 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:21.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:21 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:21.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:21 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-22T05:41:21.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:21 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:41:22.075 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:41:21 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[60807]: ts=2024-03-22T05:41:21.800Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=5 2024-03-22T05:41:22.075 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:41:21 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[60807]: ts=2024-03-22T05:41:21.803Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=4 maxSegment=5 2024-03-22T05:41:22.075 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:41:21 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[60807]: ts=2024-03-22T05:41:21.803Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=5 maxSegment=5 2024-03-22T05:41:22.075 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:41:21 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[60807]: ts=2024-03-22T05:41:21.803Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=44.019µs wal_replay_duration=725.936201ms total_replay_duration=725.993726ms 2024-03-22T05:41:22.075 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:41:21 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[60807]: ts=2024-03-22T05:41:21.808Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-03-22T05:41:22.075 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:41:21 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[60807]: ts=2024-03-22T05:41:21.808Z caller=main.go:947 level=info msg="TSDB started" 2024-03-22T05:41:22.075 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:41:21 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[60807]: ts=2024-03-22T05:41:21.808Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-22T05:41:22.075 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:41:21 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[60807]: ts=2024-03-22T05:41:21.828Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=20.265679ms db_storage=659ns remote_storage=1.414µs web_handler=329ns query_engine=882ns scrape=470.097µs scrape_sd=109.05µs notify=19.349µs notify_sd=15.381µs rules=19.296088ms 2024-03-22T05:41:22.075 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:41:21 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[60807]: ts=2024-03-22T05:41:21.828Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-03-22T05:41:22.779 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:22 smithi186 ceph-mon[25720]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-22T05:41:22.779 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:22 smithi186 ceph-mon[25720]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:41:22.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:22 smithi005 ceph-mon[24793]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-22T05:41:22.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:22 smithi005 ceph-mon[24793]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:41:22.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:22 smithi005 ceph-mon[29523]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-22T05:41:22.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:22 smithi005 ceph-mon[29523]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:41:23.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:23 smithi005 ceph-mon[29523]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 77 op/s 2024-03-22T05:41:23.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:23 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:41:23.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:23 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-22T05:41:23.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:23 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-22T05:41:23.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:23 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-22T05:41:23.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:23 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:41:23.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:23 smithi005 ceph-mon[24793]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 77 op/s 2024-03-22T05:41:23.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:23 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:41:23.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:23 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-22T05:41:23.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:23 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-22T05:41:23.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:23 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-22T05:41:23.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:23 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:41:24.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:23 smithi186 ceph-mon[25720]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s rd, 0 B/s wr, 77 op/s 2024-03-22T05:41:24.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:23 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:41:24.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:23 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-22T05:41:24.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:23 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-22T05:41:24.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:23 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-22T05:41:24.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:23 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:41:24.732 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:24 smithi186 ceph-mon[25720]: Upgrade: Updating mgr.x 2024-03-22T05:41:24.732 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:24 smithi186 ceph-mon[25720]: Deploying daemon mgr.x on smithi186 2024-03-22T05:41:24.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:24 smithi005 ceph-mon[29523]: Upgrade: Updating mgr.x 2024-03-22T05:41:24.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:24 smithi005 ceph-mon[29523]: Deploying daemon mgr.x on smithi186 2024-03-22T05:41:24.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:24 smithi005 ceph-mon[24793]: Upgrade: Updating mgr.x 2024-03-22T05:41:24.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:24 smithi005 ceph-mon[24793]: Deploying daemon mgr.x on smithi186 2024-03-22T05:41:25.030 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:41:24 smithi186 systemd[1]: Stopping Ceph mgr.x for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:41:25.293 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:41:25 smithi186 podman[61145]: 2024-03-22 05:41:25.030613373 +0000 UTC m=+0.156587529 container died 66924e151e7b8e710dc358650ded703b4f0c25a546259214b18927ca7133ea56 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217) 2024-03-22T05:41:25.633 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:41:25 smithi186 podman[61145]: 2024-03-22 05:41:25.458318804 +0000 UTC m=+0.584292955 container cleanup 66924e151e7b8e710dc358650ded703b4f0c25a546259214b18927ca7133ea56 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-22T05:41:25.633 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:41:25 smithi186 bash[61145]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x 2024-03-22T05:41:25.888 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:25 smithi186 ceph-mon[25720]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 99 KiB/s rd, 0 B/s wr, 143 op/s 2024-03-22T05:41:25.888 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:41:25 smithi186 podman[61159]: 2024-03-22 05:41:25.634548633 +0000 UTC m=+0.602297091 container remove 66924e151e7b8e710dc358650ded703b4f0c25a546259214b18927ca7133ea56 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-22T05:41:25.888 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:41:25 smithi186 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-03-22T05:41:25.888 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:41:25 smithi186 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@mgr.x.service: Failed with result 'exit-code'. 2024-03-22T05:41:25.888 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:41:25 smithi186 systemd[1]: Stopped Ceph mgr.x for f59f36e2-e80a-11ee-95cd-87774f69a715. 2024-03-22T05:41:25.888 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:41:25 smithi186 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@mgr.x.service: Consumed 58.249s CPU time. 2024-03-22T05:41:25.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:25 smithi005 ceph-mon[24793]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 99 KiB/s rd, 0 B/s wr, 143 op/s 2024-03-22T05:41:25.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:25 smithi005 ceph-mon[29523]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 99 KiB/s rd, 0 B/s wr, 143 op/s 2024-03-22T05:41:26.284 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:41:26 smithi186 systemd[1]: Starting Ceph mgr.x for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:41:26.544 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:41:26 smithi186 podman[61293]: 2024-03-22 05:41:26.187876677 +0000 UTC m=+0.017871375 image pull aa461e7c5e7f322b169c789b389d594b994ea7ffcbc9e24e563a3ec9b69bc3c1 quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9 2024-03-22T05:41:26.544 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:41:26 smithi186 podman[61293]: 2024-03-22 05:41:26.302376529 +0000 UTC m=+0.132371260 container create c556421a73f3781265649ad0e2056caac1cc5811fe40cfbecef68766d755c7b3 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-22T05:41:26.544 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:41:26 smithi186 podman[61293]: 2024-03-22 05:41:26.521705283 +0000 UTC m=+0.351699978 container init c556421a73f3781265649ad0e2056caac1cc5811fe40cfbecef68766d755c7b3 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-22T05:41:26.544 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:41:26 smithi186 podman[61293]: 2024-03-22 05:41:26.524628534 +0000 UTC m=+0.354623220 container start c556421a73f3781265649ad0e2056caac1cc5811fe40cfbecef68766d755c7b3 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-22T05:41:27.043 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:41:26 smithi186 ceph-mgr[61313]: -- 172.21.15.186:0/1362186217 <== mon.2 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5600ebf265a0 con 0x5600ebefa800 2024-03-22T05:41:27.043 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:41:26 smithi186 bash[61293]: c556421a73f3781265649ad0e2056caac1cc5811fe40cfbecef68766d755c7b3 2024-03-22T05:41:27.043 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:41:26 smithi186 systemd[1]: Started Ceph mgr.x for f59f36e2-e80a-11ee-95cd-87774f69a715. 2024-03-22T05:41:27.452 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:26 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:41:26] "GET /metrics HTTP/1.1" 200 37978 "" "Prometheus/2.33.4" 2024-03-22T05:41:27.518 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:41:27 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[61309]: 2024-03-22T05:41:27.091+0000 7fd001207200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-22T05:41:27.519 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:41:27 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[61309]: 2024-03-22T05:41:27.203+0000 7fd001207200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-22T05:41:27.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:27 smithi186 ceph-mon[25720]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-03-22T05:41:27.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:27 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:27.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:27 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:27.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:27 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:41:27.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:27 smithi005 ceph-mon[29523]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-03-22T05:41:27.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:27 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:27.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:27 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:27.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:27 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:41:27.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:27 smithi005 ceph-mon[24793]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-03-22T05:41:27.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:27 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:27.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:27 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:27.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:27 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:41:28.142 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:41:27 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[61309]: 2024-03-22T05:41:27.886+0000 7fd001207200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-22T05:41:28.142 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:41:27 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[61309]: 2024-03-22T05:41:27.992+0000 7fd001207200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-22T05:41:28.407 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:41:28 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[61309]: 2024-03-22T05:41:28.142+0000 7fd001207200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-22T05:41:28.407 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:41:28 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[61309]: 2024-03-22T05:41:28.239+0000 7fd001207200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-22T05:41:28.407 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:41:28 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[61309]: 2024-03-22T05:41:28.326+0000 7fd001207200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-22T05:41:28.714 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:41:28 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[61309]: 2024-03-22T05:41:28.602+0000 7fd001207200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-22T05:41:29.043 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:41:28 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[61309]: 2024-03-22T05:41:28.715+0000 7fd001207200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-22T05:41:29.043 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:41:28 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[61309]: 2024-03-22T05:41:28.819+0000 7fd001207200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-22T05:41:29.043 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:41:28 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[61309]: 2024-03-22T05:41:28.934+0000 7fd001207200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-22T05:41:29.393 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:41:29 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[61309]: 2024-03-22T05:41:29.135+0000 7fd001207200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-22T05:41:29.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:29 smithi005 ceph-mon[24793]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-03-22T05:41:29.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:29 smithi005 ceph-mon[29523]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-03-22T05:41:29.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:29 smithi186 ceph-mon[25720]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-03-22T05:41:30.543 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:41:30 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[61309]: 2024-03-22T05:41:30.178+0000 7fd001207200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-22T05:41:30.543 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:41:30 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[61309]: 2024-03-22T05:41:30.312+0000 7fd001207200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-22T05:41:31.004 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:41:30 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[61309]: 2024-03-22T05:41:30.697+0000 7fd001207200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-22T05:41:31.004 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:41:30 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[61309]: 2024-03-22T05:41:30.794+0000 7fd001207200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-22T05:41:31.004 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:41:31 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[61309]: 2024-03-22T05:41:31.004+0000 7fd001207200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-22T05:41:31.293 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:41:31 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[61309]: 2024-03-22T05:41:31.210+0000 7fd001207200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-22T05:41:31.793 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:41:31 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[61309]: 2024-03-22T05:41:31.520+0000 7fd001207200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-22T05:41:31.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:31 smithi186 ceph-mon[25720]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-03-22T05:41:31.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:31 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:31.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:31 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:31.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:31 smithi005 ceph-mon[29523]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-03-22T05:41:31.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:31 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:31.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:31 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:31.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:31 smithi005 ceph-mon[24793]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-03-22T05:41:31.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:31 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:31.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:31 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:32.052 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:41:31 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[61309]: 2024-03-22T05:41:31.795+0000 7fd001207200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-22T05:41:32.449 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:41:32 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[61309]: 2024-03-22T05:41:32.053+0000 7fd001207200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-22T05:41:32.758 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:32 smithi186 ceph-mon[25720]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:41:32.758 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:32 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:32.758 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:32 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:32.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:32 smithi005 ceph-mon[24793]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:41:32.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:32 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:32.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:32 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:32.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:32 smithi005 ceph-mon[29523]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:41:32.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:32 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:32.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:32 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:33.043 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:41:32 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[61309]: 2024-03-22T05:41:32.914+0000 7fd001207200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-22T05:41:33.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:33 smithi186 ceph-mon[25720]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-22T05:41:33.794 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:41:33 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[61309]: 2024-03-22T05:41:33.509+0000 7fd001207200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-22T05:41:33.794 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:41:33 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[61309]: 2024-03-22T05:41:33.620+0000 7fd001207200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-22T05:41:33.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:33 smithi005 ceph-mon[29523]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-22T05:41:33.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:33 smithi005 ceph-mon[24793]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-22T05:41:34.294 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:41:33 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[61309]: 2024-03-22T05:41:33.852+0000 7fd001207200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-22T05:41:34.294 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:41:33 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[61309]: [22/Mar/2024:05:41:33] ENGINE Bus STARTING 2024-03-22T05:41:34.294 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:41:33 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[61309]: CherryPy Checker: 2024-03-22T05:41:34.294 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:41:33 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[61309]: The Application mounted at '' has an empty config. 2024-03-22T05:41:34.294 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:41:33 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[61309]: 2024-03-22T05:41:34.294 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:41:33 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[61309]: [22/Mar/2024:05:41:33] ENGINE Serving on http://:::9283 2024-03-22T05:41:34.294 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:41:33 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[61309]: [22/Mar/2024:05:41:33] ENGINE Bus STARTED 2024-03-22T05:41:34.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:34 smithi005 ceph-mon[29523]: Standby manager daemon x restarted 2024-03-22T05:41:34.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:34 smithi005 ceph-mon[29523]: Standby manager daemon x started 2024-03-22T05:41:34.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:34 smithi005 ceph-mon[29523]: from='mgr.? 172.21.15.186:0/1878420715' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-22T05:41:34.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:34 smithi005 ceph-mon[29523]: from='mgr.? 172.21.15.186:0/1878420715' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-22T05:41:34.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:34 smithi005 ceph-mon[29523]: from='mgr.? 172.21.15.186:0/1878420715' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-22T05:41:34.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:34 smithi005 ceph-mon[29523]: from='mgr.? 172.21.15.186:0/1878420715' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-22T05:41:34.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:34 smithi005 ceph-mon[24793]: Standby manager daemon x restarted 2024-03-22T05:41:34.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:34 smithi005 ceph-mon[24793]: Standby manager daemon x started 2024-03-22T05:41:34.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:34 smithi005 ceph-mon[24793]: from='mgr.? 172.21.15.186:0/1878420715' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-22T05:41:34.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:34 smithi005 ceph-mon[24793]: from='mgr.? 172.21.15.186:0/1878420715' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-22T05:41:34.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:34 smithi005 ceph-mon[24793]: from='mgr.? 172.21.15.186:0/1878420715' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-22T05:41:34.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:34 smithi005 ceph-mon[24793]: from='mgr.? 172.21.15.186:0/1878420715' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-22T05:41:34.971 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:34 smithi186 ceph-mon[25720]: Standby manager daemon x restarted 2024-03-22T05:41:34.971 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:34 smithi186 ceph-mon[25720]: Standby manager daemon x started 2024-03-22T05:41:34.971 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:34 smithi186 ceph-mon[25720]: from='mgr.? 172.21.15.186:0/1878420715' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-22T05:41:34.971 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:34 smithi186 ceph-mon[25720]: from='mgr.? 172.21.15.186:0/1878420715' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-22T05:41:34.971 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:34 smithi186 ceph-mon[25720]: from='mgr.? 172.21.15.186:0/1878420715' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-22T05:41:34.971 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:34 smithi186 ceph-mon[25720]: from='mgr.? 172.21.15.186:0/1878420715' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-22T05:41:35.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:35 smithi005 ceph-mon[24793]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-22T05:41:35.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:35 smithi005 ceph-mon[24793]: mgrmap e43: y(active, since 27s), standbys: x 2024-03-22T05:41:35.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:35 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:35.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:35 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:35.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:35 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:41:35.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:35 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:41:35.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:35 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:35.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:35 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:41:35.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:35 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:41:35.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:35 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:41:35.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:35 smithi005 ceph-mon[29523]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-22T05:41:35.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:35 smithi005 ceph-mon[29523]: mgrmap e43: y(active, since 27s), standbys: x 2024-03-22T05:41:35.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:35 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:35.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:35 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:35.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:35 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:41:35.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:35 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:41:35.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:35 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:35.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:35 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:41:35.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:35 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:41:35.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:35 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:41:36.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:35 smithi186 ceph-mon[25720]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-22T05:41:36.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:35 smithi186 ceph-mon[25720]: mgrmap e43: y(active, since 27s), standbys: x 2024-03-22T05:41:36.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:35 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:36.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:35 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:36.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:35 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:41:36.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:35 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:41:36.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:35 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:36.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:35 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:41:36.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:35 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:41:36.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:35 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:41:36.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:36 smithi005 ceph-mon[24793]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 952 B/s rd, 0 op/s 2024-03-22T05:41:36.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:36 smithi005 ceph-mon[24793]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:41:36.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:36 smithi005 ceph-mon[24793]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:41:36.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:36 smithi005 ceph-mon[24793]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:41:36.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:36 smithi005 ceph-mon[29523]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 952 B/s rd, 0 op/s 2024-03-22T05:41:36.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:36 smithi005 ceph-mon[29523]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:41:36.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:36 smithi005 ceph-mon[29523]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:41:36.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:36 smithi005 ceph-mon[29523]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:41:37.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:36 smithi186 ceph-mon[25720]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 952 B/s rd, 0 op/s 2024-03-22T05:41:37.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:36 smithi186 ceph-mon[25720]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:41:37.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:36 smithi186 ceph-mon[25720]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:41:37.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:36 smithi186 ceph-mon[25720]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:41:37.320 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:36 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:41:36] "GET /metrics HTTP/1.1" 200 37978 "" "Prometheus/2.33.4" 2024-03-22T05:41:37.683 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:41:37.467+0000 7f487c563700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:41:37.684 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:41:37.684 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:41:37.684 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:41:37.684 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:41:37.684 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:41:37.684 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:41:37.684 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:41:37.684 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:41:37.684 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:41:37.684 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:41:37.684 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:41:37.684 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:41:37.684 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:41:37.684 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:41:37.684 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:41:37.685 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:41:37.685 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:41:37.685 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:41:37.685 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:41:37.685 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:41:37.685 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:37.685 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-03-22T05:41:37.685 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-03-22T05:41:37.686 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-03-22T05:41:37.686 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-03-22T05:41:37.686 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-03-22T05:41:37.686 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-03-22T05:41:37.686 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:41:37.686 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:41:37.686 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:41:37.686 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:37.686 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-03-22T05:41:37.687 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-03-22T05:41:37.687 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-03-22T05:41:37.687 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-03-22T05:41:37.687 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-03-22T05:41:37.687 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-03-22T05:41:37.687 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:41:37.687 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:41:37.687 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:41:37.688 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:41:37.688 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:41:37.688 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:41:37.688 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:41:37.688 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:41:37.688 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:41:37.688 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:41:37.688 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:41:37.688 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return future.result(timeout) 2024-03-22T05:41:37.688 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:41:37.688 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.__get_result() 2024-03-22T05:41:37.688 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:41:37.688 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: raise self._exception 2024-03-22T05:41:37.688 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:41:37.688 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: config_blobs=daemon_spec.final_config, 2024-03-22T05:41:37.689 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:41:37.689 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:41:37.689 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:41:37.689 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:41:37.689 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:41:37.689 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:41:37.689 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:41:37.689 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:41:37.689 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:41:37.689 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:41:37.689 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:41:37.689 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:41:37.689 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:41:37.689 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:41:37.690 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:41:37.690 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:41:37.690 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:41:37.690 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:41:37.690 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:41:37.690 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:41:37.690 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:41:37.690 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:41:37.690 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:41:37.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:37.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:41:37.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:37.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:41:37.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:37.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:41:37.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:37.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:41:37.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:37.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:41:37.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:37.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:41:37.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:37.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:37.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:41:37.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:37.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:41:37.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:37.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:41:37.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:37.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:41:37.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:37.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:41:37.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:37.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:41:37.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:37 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:38.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:37 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:41:38.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:37 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:41:38.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:37 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:38.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:37 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-03-22T05:41:38.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:37 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-03-22T05:41:38.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:37 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-03-22T05:41:38.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:37 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-03-22T05:41:38.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:37 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-03-22T05:41:38.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:37 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-03-22T05:41:38.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:37 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:41:38.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:37 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:38.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:37 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:41:38.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:37 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:38.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:37 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:41:38.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:37 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:38.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:37 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:41:38.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:37 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:38.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:37 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:41:38.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:37 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:38.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:37 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:41:38.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:37 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:38.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:37 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:41:38.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:37 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:38.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:38 smithi005 ceph-mon[24793]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:41:38.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:38 smithi005 ceph-mon[24793]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:41:38.952 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:41:38.952 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:41:38.952 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:41:38.952 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:41:38.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:41:38.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:41:38.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:41:38.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:41:38.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:41:38.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:41:38.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:41:38.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:41:38.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:41:38.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:41:38.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:41:38.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:41:38.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:41:38.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:41:38.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:41:38.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:41:38.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:41:38.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:41:38.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:41:38.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:41:38.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:41:38.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:41:38.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:41:38.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:41:38.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: return future.result(timeout) 2024-03-22T05:41:38.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:41:38.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.__get_result() 2024-03-22T05:41:38.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:41:38.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: raise self._exception 2024-03-22T05:41:38.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:41:38.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:41:38.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:41:38.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:41:38.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:41:38.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:41:38.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:41:38.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:41:38.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:41:38.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:41:38.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:41:38.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:41:38.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:41:38.956 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:41:38.956 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:41:38.956 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:41:38.956 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:41:38.956 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:41:38.956 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:41:38.956 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:41:38.956 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:38 smithi005 ceph-mon[29523]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:41:38.956 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:38 smithi005 ceph-mon[29523]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:41:38.956 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:41:38.956 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:41:38.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:41:38.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:41:38.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:41:38.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:41:38.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:41:38.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:41:38.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:41:38.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:41:38.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:41:38.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:41:38.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:41:38.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:41:38.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:41:38.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:41:38.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:41:38.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:41:38.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:41:38.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:41:38.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:41:38.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:41:38.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:41:38.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:41:38.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:41:38.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:41:38.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:41:38.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:41:38.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: return future.result(timeout) 2024-03-22T05:41:38.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:41:38.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.__get_result() 2024-03-22T05:41:38.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:41:38.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: raise self._exception 2024-03-22T05:41:38.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:41:38.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:41:38.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:41:38.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:41:38.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:41:38.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:41:38.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:41:38.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:41:38.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:41:38.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:41:38.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:41:38.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:41:38.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:41:38.960 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:41:38.960 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:41:38.960 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:41:38.960 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:41:38.960 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:41:38.960 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:41:38.960 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:41:38.960 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:41:38.960 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:41:38.961 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:41:38.961 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:41:38.961 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:41:38.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:38 smithi005 ceph-mon[24793]: Upgrade: Setting container_image for all mgr 2024-03-22T05:41:38.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:38 smithi005 ceph-mon[24793]: Upgrade: Setting container_image for all crash 2024-03-22T05:41:38.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:38 smithi005 ceph-mon[24793]: Upgrade: Setting container_image for all mds 2024-03-22T05:41:38.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:38 smithi005 ceph-mon[24793]: Upgrade: Setting container_image for all rbd-mirror 2024-03-22T05:41:38.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:38 smithi005 ceph-mon[24793]: Upgrade: Setting container_image for all cephfs-mirror 2024-03-22T05:41:38.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:38 smithi005 ceph-mon[24793]: Upgrade: Setting container_image for all ceph-exporter 2024-03-22T05:41:38.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:38 smithi005 ceph-mon[24793]: Upgrade: Setting container_image for all nfs 2024-03-22T05:41:38.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:38 smithi005 ceph-mon[24793]: Upgrade: Setting container_image for all nvmeof 2024-03-22T05:41:38.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:38 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:41:38.961 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:41:38.962 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:41:38.962 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:41:38.962 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:41:38.962 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:41:38.962 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:38 smithi005 ceph-mon[29523]: Upgrade: Setting container_image for all mgr 2024-03-22T05:41:38.962 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:38 smithi005 ceph-mon[29523]: Upgrade: Setting container_image for all crash 2024-03-22T05:41:38.962 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:38 smithi005 ceph-mon[29523]: Upgrade: Setting container_image for all mds 2024-03-22T05:41:38.962 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:38 smithi005 ceph-mon[29523]: Upgrade: Setting container_image for all rbd-mirror 2024-03-22T05:41:38.962 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:38 smithi005 ceph-mon[29523]: Upgrade: Setting container_image for all cephfs-mirror 2024-03-22T05:41:38.962 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:38 smithi005 ceph-mon[29523]: Upgrade: Setting container_image for all ceph-exporter 2024-03-22T05:41:38.962 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:38 smithi005 ceph-mon[29523]: Upgrade: Setting container_image for all nfs 2024-03-22T05:41:38.962 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:38 smithi005 ceph-mon[29523]: Upgrade: Setting container_image for all nvmeof 2024-03-22T05:41:38.962 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:38 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:41:39.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:38 smithi186 ceph-mon[25720]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:41:39.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:38 smithi186 ceph-mon[25720]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:41:39.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:41:39.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:41:39.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:41:39.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:41:39.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:41:39.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:41:39.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:41:39.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:41:39.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:41:39.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:41:39.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:41:39.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:41:39.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:41:39.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:41:39.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:41:39.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:41:39.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:41:39.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:41:39.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:41:39.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:41:39.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:41:39.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:41:39.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:41:39.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:41:39.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:41:39.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:41:39.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:41:39.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:41:39.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: return future.result(timeout) 2024-03-22T05:41:39.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:41:39.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.__get_result() 2024-03-22T05:41:39.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:41:39.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: raise self._exception 2024-03-22T05:41:39.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:41:39.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:41:39.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:41:39.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:41:39.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:41:39.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:41:39.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:41:39.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:41:39.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:41:39.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:41:39.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:41:39.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:41:39.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:41:39.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:41:39.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:41:39.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:41:39.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:41:39.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:41:39.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:41:39.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:41:39.048 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:41:39.048 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:41:39.048 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:41:39.048 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:41:39.048 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:41:39.048 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:38 smithi186 ceph-mon[25720]: Upgrade: Setting container_image for all mgr 2024-03-22T05:41:39.048 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:38 smithi186 ceph-mon[25720]: Upgrade: Setting container_image for all crash 2024-03-22T05:41:39.048 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:38 smithi186 ceph-mon[25720]: Upgrade: Setting container_image for all mds 2024-03-22T05:41:39.048 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:38 smithi186 ceph-mon[25720]: Upgrade: Setting container_image for all rbd-mirror 2024-03-22T05:41:39.048 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:38 smithi186 ceph-mon[25720]: Upgrade: Setting container_image for all cephfs-mirror 2024-03-22T05:41:39.048 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:38 smithi186 ceph-mon[25720]: Upgrade: Setting container_image for all ceph-exporter 2024-03-22T05:41:39.048 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:38 smithi186 ceph-mon[25720]: Upgrade: Setting container_image for all nfs 2024-03-22T05:41:39.048 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:38 smithi186 ceph-mon[25720]: Upgrade: Setting container_image for all nvmeof 2024-03-22T05:41:39.048 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:38 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:41:39.952 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:41:39 smithi005 systemd[1]: Stopping Ceph node-exporter.a for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:41:40.219 INFO:teuthology.orchestra.run.smithi005.stdout:true 2024-03-22T05:41:40.408 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:41:40 smithi005 podman[75989]: 2024-03-22 05:41:40.101096567 +0000 UTC m=+0.111187539 container died a9085d6042eb06fd812007f7f7bf20615c23c8f5be96a5908548bb8b5139cbc1 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-22T05:41:40.687 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:41:40 smithi005 podman[75989]: 2024-03-22 05:41:40.515286904 +0000 UTC m=+0.525377878 container cleanup a9085d6042eb06fd812007f7f7bf20615c23c8f5be96a5908548bb8b5139cbc1 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-22T05:41:40.687 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:41:40 smithi005 bash[75989]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a 2024-03-22T05:41:40.917 INFO:teuthology.orchestra.run.smithi005.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T05:41:40.918 INFO:teuthology.orchestra.run.smithi005.stdout:alertmanager.a smithi005 *:9093,9094 running (13m) 28s ago 21m 22.8M - 0.23.0 ba2b418f427c fd7dc5dbe3e5 2024-03-22T05:41:40.918 INFO:teuthology.orchestra.run.smithi005.stdout:grafana.a smithi186 *:3000 error 10s ago 20m - - 2024-03-22T05:41:40.918 INFO:teuthology.orchestra.run.smithi005.stdout:iscsi.foo.smithi005.mcdwbp smithi005 error 28s ago 20m 58.6M - 3.5 e1d6a67b021e 5215e03bf81b 2024-03-22T05:41:40.918 INFO:teuthology.orchestra.run.smithi005.stdout:mgr.x smithi186 *:8443,9283,8765 running (14s) 10s ago 25m 64.4M - 19.0.0-2299-gea01217f aa461e7c5e7f c556421a73f3 2024-03-22T05:41:40.918 INFO:teuthology.orchestra.run.smithi005.stdout:mgr.y smithi005 *:8443,9283,8765 running (10m) 28s ago 27m 475M - 19.0.0-2299-gea01217f aa461e7c5e7f 62a813eebc0d 2024-03-22T05:41:40.918 INFO:teuthology.orchestra.run.smithi005.stdout:mon.a smithi005 running (27m) 28s ago 27m 76.4M 2048M 17.2.0 e1d6a67b021e 38e0bba4bb03 2024-03-22T05:41:40.918 INFO:teuthology.orchestra.run.smithi005.stdout:mon.b smithi186 running (25m) 10s ago 25m 54.1M 2048M 17.2.0 e1d6a67b021e 323f53f932f6 2024-03-22T05:41:40.918 INFO:teuthology.orchestra.run.smithi005.stdout:mon.c smithi005 running (26m) 28s ago 26m 55.8M 2048M 17.2.0 e1d6a67b021e 00ec24d38612 2024-03-22T05:41:40.918 INFO:teuthology.orchestra.run.smithi005.stdout:node-exporter.a smithi005 *:9100 running (13m) 28s ago 21m 16.1M - 1.3.1 1dbe0e931976 a9085d6042eb 2024-03-22T05:41:40.918 INFO:teuthology.orchestra.run.smithi005.stdout:node-exporter.b smithi186 *:9100 running (13m) 10s ago 21m 18.2M - 1.3.1 1dbe0e931976 74aa1181ff71 2024-03-22T05:41:40.918 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0 smithi005 running (25m) 28s ago 25m 52.7M 2993M 17.2.0 e1d6a67b021e ef5c7053dd37 2024-03-22T05:41:40.918 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1 smithi005 running (24m) 28s ago 24m 55.1M 2993M 17.2.0 e1d6a67b021e a1e7ee1017b6 2024-03-22T05:41:40.919 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2 smithi005 running (24m) 28s ago 24m 48.2M 2993M 17.2.0 e1d6a67b021e 462ea63ad0e5 2024-03-22T05:41:40.919 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3 smithi005 running (24m) 28s ago 24m 55.3M 2993M 17.2.0 e1d6a67b021e b25432a7239b 2024-03-22T05:41:40.919 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4 smithi186 running (23m) 10s ago 23m 56.0M 3248M 17.2.0 e1d6a67b021e cd963671f340 2024-03-22T05:41:40.919 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5 smithi186 running (23m) 10s ago 23m 51.4M 3248M 17.2.0 e1d6a67b021e f98c87f1a2cd 2024-03-22T05:41:40.919 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6 smithi186 running (23m) 10s ago 23m 51.5M 3248M 17.2.0 e1d6a67b021e 704ac49c68c4 2024-03-22T05:41:40.919 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7 smithi186 running (22m) 10s ago 22m 54.0M 3248M 17.2.0 e1d6a67b021e f415b90edf63 2024-03-22T05:41:40.919 INFO:teuthology.orchestra.run.smithi005.stdout:prometheus.a smithi186 *:9095 running (20s) 10s ago 21m 58.5M - 2.33.4 514e6a882f6e 8596d431e557 2024-03-22T05:41:40.919 INFO:teuthology.orchestra.run.smithi005.stdout:rgw.foo.smithi005.exdhqu smithi005 *:8000 running (20m) 28s ago 20m 78.2M - 17.2.0 e1d6a67b021e 6bb27fe11553 2024-03-22T05:41:40.919 INFO:teuthology.orchestra.run.smithi005.stdout:rgw.foo.smithi186.mcgodk smithi186 *:8000 running (20m) 10s ago 20m 76.5M - 17.2.0 e1d6a67b021e 19b5c20a34f7 2024-03-22T05:41:40.952 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:41:40 smithi005 podman[76004]: 2024-03-22 05:41:40.766765992 +0000 UTC m=+0.666649051 container remove a9085d6042eb06fd812007f7f7bf20615c23c8f5be96a5908548bb8b5139cbc1 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-22T05:41:40.952 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:41:40 smithi005 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-03-22T05:41:40.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:40 smithi005 ceph-mon[29523]: Upgrade: Updating node-exporter.a (1/2) 2024-03-22T05:41:40.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:40 smithi005 ceph-mon[29523]: Deploying daemon node-exporter.a on smithi005 2024-03-22T05:41:40.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:40 smithi005 ceph-mon[29523]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:41:40.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:40 smithi005 ceph-mon[24793]: Upgrade: Updating node-exporter.a (1/2) 2024-03-22T05:41:40.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:40 smithi005 ceph-mon[24793]: Deploying daemon node-exporter.a on smithi005 2024-03-22T05:41:40.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:40 smithi005 ceph-mon[24793]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:41:41.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:40 smithi186 ceph-mon[25720]: Upgrade: Updating node-exporter.a (1/2) 2024-03-22T05:41:41.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:40 smithi186 ceph-mon[25720]: Deploying daemon node-exporter.a on smithi005 2024-03-22T05:41:41.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:40 smithi186 ceph-mon[25720]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:41:41.355 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-03-22T05:41:41.355 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": { 2024-03-22T05:41:41.355 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-03-22T05:41:41.355 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:41:41.355 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr": { 2024-03-22T05:41:41.355 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev)": 2 2024-03-22T05:41:41.355 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:41:41.355 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": { 2024-03-22T05:41:41.355 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-22T05:41:41.355 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:41:41.355 INFO:teuthology.orchestra.run.smithi005.stdout: "mds": {}, 2024-03-22T05:41:41.356 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw": { 2024-03-22T05:41:41.356 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-22T05:41:41.356 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:41:41.356 INFO:teuthology.orchestra.run.smithi005.stdout: "overall": { 2024-03-22T05:41:41.356 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-03-22T05:41:41.356 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev)": 2 2024-03-22T05:41:41.356 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-03-22T05:41:41.356 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-03-22T05:41:41.356 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:41:41 smithi005 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-03-22T05:41:41.356 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:41:41 smithi005 systemd[1]: Stopped Ceph node-exporter.a for f59f36e2-e80a-11ee-95cd-87774f69a715. 2024-03-22T05:41:41.356 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:41:41 smithi005 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@node-exporter.a.service: Consumed 5.070s CPU time. 2024-03-22T05:41:41.356 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:41:41 smithi005 systemd[1]: Starting Ceph node-exporter.a for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:41:41.691 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:41:41 smithi005 bash[76205]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-03-22T05:41:41.705 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-03-22T05:41:41.705 INFO:teuthology.orchestra.run.smithi005.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9", 2024-03-22T05:41:41.705 INFO:teuthology.orchestra.run.smithi005.stdout: "in_progress": true, 2024-03-22T05:41:41.705 INFO:teuthology.orchestra.run.smithi005.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-03-22T05:41:41.705 INFO:teuthology.orchestra.run.smithi005.stdout: "services_complete": [ 2024-03-22T05:41:41.705 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr" 2024-03-22T05:41:41.705 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-03-22T05:41:41.705 INFO:teuthology.orchestra.run.smithi005.stdout: "progress": "2/2 daemons upgraded", 2024-03-22T05:41:41.705 INFO:teuthology.orchestra.run.smithi005.stdout: "message": "Currently upgrading node-exporter daemons", 2024-03-22T05:41:41.705 INFO:teuthology.orchestra.run.smithi005.stdout: "is_paused": false 2024-03-22T05:41:41.706 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-03-22T05:41:41.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:41 smithi005 ceph-mon[29523]: from='client.15462 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:41:41.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:41 smithi005 ceph-mon[29523]: from='client.15468 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:41:41.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:41 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/2470308484' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:41:41.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:41 smithi005 ceph-mon[24793]: from='client.15462 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:41:41.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:41 smithi005 ceph-mon[24793]: from='client.15468 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:41:41.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:41 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/2470308484' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:41:41.953 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:41:41 smithi005 bash[76205]: Getting image source signatures 2024-03-22T05:41:41.953 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:41:41 smithi005 bash[76205]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-03-22T05:41:41.953 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:41:41 smithi005 bash[76205]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-03-22T05:41:41.953 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:41:41 smithi005 bash[76205]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-22T05:41:42.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:41 smithi186 ceph-mon[25720]: from='client.15462 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:41:42.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:41 smithi186 ceph-mon[25720]: from='client.15468 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:41:42.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:41 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/2470308484' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:41:42.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:42 smithi005 ceph-mon[29523]: from='client.15474 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:41:42.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:42 smithi005 ceph-mon[29523]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:41:42.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:42 smithi005 ceph-mon[29523]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:41:42.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:42 smithi005 ceph-mon[29523]: from='client.25387 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:41:42.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:42 smithi005 ceph-mon[24793]: from='client.15474 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:41:42.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:42 smithi005 ceph-mon[24793]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:41:42.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:42 smithi005 ceph-mon[24793]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:41:42.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:42 smithi005 ceph-mon[24793]: from='client.25387 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:41:43.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:42 smithi186 ceph-mon[25720]: from='client.15474 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:41:43.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:42 smithi186 ceph-mon[25720]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:41:43.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:42 smithi186 ceph-mon[25720]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:41:43.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:42 smithi186 ceph-mon[25720]: from='client.25387 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:41:43.602 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:41:43 smithi005 bash[76205]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-22T05:41:43.603 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:41:43 smithi005 bash[76205]: Writing manifest to image destination 2024-03-22T05:41:43.930 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:41:43 smithi005 podman[76205]: 2024-03-22 05:41:43.729331806 +0000 UTC m=+2.409065952 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-22T05:41:43.931 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:41:43 smithi005 podman[76205]: 2024-03-22 05:41:43.827980215 +0000 UTC m=+2.507714353 container create f635cd72b9d3d88c7ae61ac71a116adc9547b806cdaaa5c33d083cccc0df5576 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-22T05:41:44.202 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:41:44 smithi005 podman[76205]: 2024-03-22 05:41:44.047223011 +0000 UTC m=+2.726957160 container init f635cd72b9d3d88c7ae61ac71a116adc9547b806cdaaa5c33d083cccc0df5576 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-22T05:41:44.202 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:41:44 smithi005 podman[76205]: 2024-03-22 05:41:44.050012551 +0000 UTC m=+2.729746692 container start f635cd72b9d3d88c7ae61ac71a116adc9547b806cdaaa5c33d083cccc0df5576 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-22T05:41:44.202 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:41:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[76292]: ts=2024-03-22T05:41:44.053Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-22T05:41:44.202 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:41:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[76292]: ts=2024-03-22T05:41:44.053Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-22T05:41:44.202 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:41:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[76292]: ts=2024-03-22T05:41:44.054Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-22T05:41:44.203 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:41:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[76292]: ts=2024-03-22T05:41:44.054Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-22T05:41:44.203 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:41:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[76292]: ts=2024-03-22T05:41:44.054Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-22T05:41:44.203 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:41:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[76292]: ts=2024-03-22T05:41:44.054Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-22T05:41:44.203 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:41:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[76292]: ts=2024-03-22T05:41:44.054Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-22T05:41:44.203 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:41:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[76292]: ts=2024-03-22T05:41:44.054Z caller=node_exporter.go:117 level=info collector=arp 2024-03-22T05:41:44.203 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:41:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[76292]: ts=2024-03-22T05:41:44.054Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-22T05:41:44.203 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:41:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[76292]: ts=2024-03-22T05:41:44.054Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-22T05:41:44.203 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:41:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[76292]: ts=2024-03-22T05:41:44.054Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-22T05:41:44.203 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:41:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[76292]: ts=2024-03-22T05:41:44.054Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-22T05:41:44.203 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:41:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[76292]: ts=2024-03-22T05:41:44.054Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-22T05:41:44.203 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:41:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[76292]: ts=2024-03-22T05:41:44.054Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-22T05:41:44.204 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:41:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[76292]: ts=2024-03-22T05:41:44.054Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-22T05:41:44.204 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:41:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[76292]: ts=2024-03-22T05:41:44.054Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-22T05:41:44.204 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:41:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[76292]: ts=2024-03-22T05:41:44.054Z caller=node_exporter.go:117 level=info collector=edac 2024-03-22T05:41:44.204 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:41:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[76292]: ts=2024-03-22T05:41:44.055Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-22T05:41:44.204 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:41:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[76292]: ts=2024-03-22T05:41:44.055Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-22T05:41:44.204 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:41:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[76292]: ts=2024-03-22T05:41:44.055Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-22T05:41:44.204 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:41:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[76292]: ts=2024-03-22T05:41:44.055Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-22T05:41:44.204 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:41:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[76292]: ts=2024-03-22T05:41:44.055Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-22T05:41:44.204 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:41:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[76292]: ts=2024-03-22T05:41:44.055Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-22T05:41:44.204 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:41:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[76292]: ts=2024-03-22T05:41:44.055Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-22T05:41:44.204 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:41:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[76292]: ts=2024-03-22T05:41:44.055Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-22T05:41:44.205 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:41:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[76292]: ts=2024-03-22T05:41:44.055Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-22T05:41:44.205 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:41:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[76292]: ts=2024-03-22T05:41:44.055Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-22T05:41:44.205 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:41:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[76292]: ts=2024-03-22T05:41:44.055Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-22T05:41:44.205 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:41:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[76292]: ts=2024-03-22T05:41:44.055Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-22T05:41:44.205 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:41:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[76292]: ts=2024-03-22T05:41:44.055Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-22T05:41:44.205 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:41:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[76292]: ts=2024-03-22T05:41:44.055Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-22T05:41:44.205 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:41:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[76292]: ts=2024-03-22T05:41:44.055Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-22T05:41:44.205 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:41:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[76292]: ts=2024-03-22T05:41:44.055Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-22T05:41:44.205 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:41:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[76292]: ts=2024-03-22T05:41:44.055Z caller=node_exporter.go:117 level=info collector=os 2024-03-22T05:41:44.205 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:41:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[76292]: ts=2024-03-22T05:41:44.055Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-22T05:41:44.205 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:41:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[76292]: ts=2024-03-22T05:41:44.055Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-22T05:41:44.205 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:41:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[76292]: ts=2024-03-22T05:41:44.055Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-22T05:41:44.205 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:41:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[76292]: ts=2024-03-22T05:41:44.055Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-22T05:41:44.205 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:41:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[76292]: ts=2024-03-22T05:41:44.055Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-22T05:41:44.205 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:41:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[76292]: ts=2024-03-22T05:41:44.055Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-22T05:41:44.206 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:41:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[76292]: ts=2024-03-22T05:41:44.055Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-22T05:41:44.206 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:41:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[76292]: ts=2024-03-22T05:41:44.055Z caller=node_exporter.go:117 level=info collector=stat 2024-03-22T05:41:44.206 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:41:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[76292]: ts=2024-03-22T05:41:44.055Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-22T05:41:44.206 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:41:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[76292]: ts=2024-03-22T05:41:44.055Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-22T05:41:44.206 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:41:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[76292]: ts=2024-03-22T05:41:44.055Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-22T05:41:44.206 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:41:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[76292]: ts=2024-03-22T05:41:44.055Z caller=node_exporter.go:117 level=info collector=time 2024-03-22T05:41:44.206 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:41:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[76292]: ts=2024-03-22T05:41:44.055Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-22T05:41:44.206 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:41:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[76292]: ts=2024-03-22T05:41:44.055Z caller=node_exporter.go:117 level=info collector=uname 2024-03-22T05:41:44.206 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:41:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[76292]: ts=2024-03-22T05:41:44.055Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-22T05:41:44.206 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:41:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[76292]: ts=2024-03-22T05:41:44.055Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-22T05:41:44.206 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:41:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[76292]: ts=2024-03-22T05:41:44.055Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-22T05:41:44.206 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:41:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[76292]: ts=2024-03-22T05:41:44.055Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-22T05:41:44.206 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:41:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a[76292]: ts=2024-03-22T05:41:44.055Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-22T05:41:44.206 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:41:44 smithi005 bash[76205]: f635cd72b9d3d88c7ae61ac71a116adc9547b806cdaaa5c33d083cccc0df5576 2024-03-22T05:41:44.207 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:41:44 smithi005 systemd[1]: Started Ceph node-exporter.a for f59f36e2-e80a-11ee-95cd-87774f69a715. 2024-03-22T05:41:44.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:44 smithi005 ceph-mon[24793]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 585 B/s rd, 0 op/s 2024-03-22T05:41:44.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:44 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:44.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:44 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:44.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:44 smithi005 ceph-mon[29523]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 585 B/s rd, 0 op/s 2024-03-22T05:41:44.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:44 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:44.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:44 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:44.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:44 smithi186 ceph-mon[25720]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 585 B/s rd, 0 op/s 2024-03-22T05:41:44.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:44 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:44.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:44 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:46.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:46 smithi186 ceph-mon[25720]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:41:46.544 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:41:46 smithi186 systemd[1]: Stopping Ceph node-exporter.b for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:41:46.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:46 smithi005 ceph-mon[24793]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:41:46.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:46 smithi005 ceph-mon[29523]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:41:46.976 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:41:46 smithi186 podman[62648]: 2024-03-22 05:41:46.696292576 +0000 UTC m=+0.118079604 container died 74aa1181ff717bd9b36ec7e639ed394fff71031a1d4ffc165c74454aad00f662 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-22T05:41:47.235 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:41:46 smithi186 podman[62648]: 2024-03-22 05:41:46.977117683 +0000 UTC m=+0.398904719 container cleanup 74aa1181ff717bd9b36ec7e639ed394fff71031a1d4ffc165c74454aad00f662 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-22T05:41:47.236 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:41:46 smithi186 bash[62648]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b 2024-03-22T05:41:47.399 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:46 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:41:46] "GET /metrics HTTP/1.1" 200 37983 "" "Prometheus/2.33.4" 2024-03-22T05:41:47.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:47 smithi186 ceph-mon[25720]: Upgrade: Updating node-exporter.b (2/2) 2024-03-22T05:41:47.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:47 smithi186 ceph-mon[25720]: Deploying daemon node-exporter.b on smithi186 2024-03-22T05:41:47.544 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:41:47 smithi186 podman[62662]: 2024-03-22 05:41:47.236801039 +0000 UTC m=+0.541684712 container remove 74aa1181ff717bd9b36ec7e639ed394fff71031a1d4ffc165c74454aad00f662 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-22T05:41:47.544 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:41:47 smithi186 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-03-22T05:41:47.544 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:41:47 smithi186 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-03-22T05:41:47.544 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:41:47 smithi186 systemd[1]: Stopped Ceph node-exporter.b for f59f36e2-e80a-11ee-95cd-87774f69a715. 2024-03-22T05:41:47.544 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:41:47 smithi186 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@node-exporter.b.service: Consumed 5.098s CPU time. 2024-03-22T05:41:47.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:47 smithi005 ceph-mon[24793]: Upgrade: Updating node-exporter.b (2/2) 2024-03-22T05:41:47.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:47 smithi005 ceph-mon[24793]: Deploying daemon node-exporter.b on smithi186 2024-03-22T05:41:47.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:47 smithi005 ceph-mon[29523]: Upgrade: Updating node-exporter.b (2/2) 2024-03-22T05:41:47.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:47 smithi005 ceph-mon[29523]: Deploying daemon node-exporter.b on smithi186 2024-03-22T05:41:48.043 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:41:47 smithi186 systemd[1]: Starting Ceph node-exporter.b for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:41:48.044 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:41:47 smithi186 bash[62797]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-03-22T05:41:48.401 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:41:48 smithi186 bash[62797]: Getting image source signatures 2024-03-22T05:41:48.401 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:41:48 smithi186 bash[62797]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-03-22T05:41:48.401 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:41:48 smithi186 bash[62797]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-22T05:41:48.401 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:41:48 smithi186 bash[62797]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-03-22T05:41:48.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:48 smithi005 ceph-mon[24793]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:41:48.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:48 smithi005 ceph-mon[29523]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:41:48.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:48 smithi186 ceph-mon[25720]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:41:50.293 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:41:49 smithi186 bash[62797]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-22T05:41:50.293 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:41:49 smithi186 bash[62797]: Writing manifest to image destination 2024-03-22T05:41:50.649 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:50 smithi186 ceph-mon[25720]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:41:50.650 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:41:50 smithi186 podman[62797]: 2024-03-22 05:41:50.249220113 +0000 UTC m=+2.544471028 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-22T05:41:50.650 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:41:50 smithi186 podman[62797]: 2024-03-22 05:41:50.356409681 +0000 UTC m=+2.651660588 container create d7e0b3d201918d57c0675d79f678adb3a4c4d5a8cc2059bf7cdffb53411aa321 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-22T05:41:50.650 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:41:50 smithi186 podman[62797]: 2024-03-22 05:41:50.567161848 +0000 UTC m=+2.862412768 container init d7e0b3d201918d57c0675d79f678adb3a4c4d5a8cc2059bf7cdffb53411aa321 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-22T05:41:50.650 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:41:50 smithi186 podman[62797]: 2024-03-22 05:41:50.569937118 +0000 UTC m=+2.865188032 container start d7e0b3d201918d57c0675d79f678adb3a4c4d5a8cc2059bf7cdffb53411aa321 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-22T05:41:50.650 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:41:50 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[62847]: ts=2024-03-22T05:41:50.573Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-22T05:41:50.650 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:41:50 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[62847]: ts=2024-03-22T05:41:50.573Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-22T05:41:50.651 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:41:50 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[62847]: ts=2024-03-22T05:41:50.574Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-22T05:41:50.651 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:41:50 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[62847]: ts=2024-03-22T05:41:50.574Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-22T05:41:50.651 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:41:50 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[62847]: ts=2024-03-22T05:41:50.574Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-22T05:41:50.651 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:41:50 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[62847]: ts=2024-03-22T05:41:50.574Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-22T05:41:50.651 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:41:50 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[62847]: ts=2024-03-22T05:41:50.574Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-22T05:41:50.651 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:41:50 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[62847]: ts=2024-03-22T05:41:50.574Z caller=node_exporter.go:117 level=info collector=arp 2024-03-22T05:41:50.651 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:41:50 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[62847]: ts=2024-03-22T05:41:50.574Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-22T05:41:50.651 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:41:50 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[62847]: ts=2024-03-22T05:41:50.574Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-22T05:41:50.651 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:41:50 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[62847]: ts=2024-03-22T05:41:50.574Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-22T05:41:50.652 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:41:50 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[62847]: ts=2024-03-22T05:41:50.574Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-22T05:41:50.652 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:41:50 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[62847]: ts=2024-03-22T05:41:50.575Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-22T05:41:50.652 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:41:50 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[62847]: ts=2024-03-22T05:41:50.575Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-22T05:41:50.652 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:41:50 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[62847]: ts=2024-03-22T05:41:50.575Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-22T05:41:50.652 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:41:50 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[62847]: ts=2024-03-22T05:41:50.575Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-22T05:41:50.652 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:41:50 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[62847]: ts=2024-03-22T05:41:50.575Z caller=node_exporter.go:117 level=info collector=edac 2024-03-22T05:41:50.652 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:41:50 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[62847]: ts=2024-03-22T05:41:50.575Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-22T05:41:50.652 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:41:50 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[62847]: ts=2024-03-22T05:41:50.575Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-22T05:41:50.653 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:41:50 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[62847]: ts=2024-03-22T05:41:50.575Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-22T05:41:50.653 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:41:50 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[62847]: ts=2024-03-22T05:41:50.575Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-22T05:41:50.653 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:41:50 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[62847]: ts=2024-03-22T05:41:50.575Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-22T05:41:50.653 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:41:50 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[62847]: ts=2024-03-22T05:41:50.575Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-22T05:41:50.653 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:41:50 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[62847]: ts=2024-03-22T05:41:50.575Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-22T05:41:50.653 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:41:50 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[62847]: ts=2024-03-22T05:41:50.575Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-22T05:41:50.653 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:41:50 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[62847]: ts=2024-03-22T05:41:50.575Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-22T05:41:50.653 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:41:50 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[62847]: ts=2024-03-22T05:41:50.575Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-22T05:41:50.654 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:41:50 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[62847]: ts=2024-03-22T05:41:50.575Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-22T05:41:50.654 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:41:50 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[62847]: ts=2024-03-22T05:41:50.575Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-22T05:41:50.654 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:41:50 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[62847]: ts=2024-03-22T05:41:50.575Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-22T05:41:50.654 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:41:50 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[62847]: ts=2024-03-22T05:41:50.575Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-22T05:41:50.654 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:41:50 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[62847]: ts=2024-03-22T05:41:50.575Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-22T05:41:50.654 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:41:50 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[62847]: ts=2024-03-22T05:41:50.575Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-22T05:41:50.654 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:41:50 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[62847]: ts=2024-03-22T05:41:50.575Z caller=node_exporter.go:117 level=info collector=os 2024-03-22T05:41:50.654 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:41:50 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[62847]: ts=2024-03-22T05:41:50.575Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-22T05:41:50.654 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:41:50 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[62847]: ts=2024-03-22T05:41:50.575Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-22T05:41:50.654 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:41:50 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[62847]: ts=2024-03-22T05:41:50.575Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-22T05:41:50.654 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:41:50 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[62847]: ts=2024-03-22T05:41:50.575Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-22T05:41:50.654 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:41:50 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[62847]: ts=2024-03-22T05:41:50.575Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-22T05:41:50.654 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:41:50 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[62847]: ts=2024-03-22T05:41:50.575Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-22T05:41:50.654 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:41:50 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[62847]: ts=2024-03-22T05:41:50.575Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-22T05:41:50.655 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:41:50 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[62847]: ts=2024-03-22T05:41:50.575Z caller=node_exporter.go:117 level=info collector=stat 2024-03-22T05:41:50.655 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:41:50 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[62847]: ts=2024-03-22T05:41:50.575Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-22T05:41:50.655 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:41:50 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[62847]: ts=2024-03-22T05:41:50.575Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-22T05:41:50.655 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:41:50 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[62847]: ts=2024-03-22T05:41:50.575Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-22T05:41:50.655 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:41:50 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[62847]: ts=2024-03-22T05:41:50.575Z caller=node_exporter.go:117 level=info collector=time 2024-03-22T05:41:50.655 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:41:50 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[62847]: ts=2024-03-22T05:41:50.575Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-22T05:41:50.655 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:41:50 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[62847]: ts=2024-03-22T05:41:50.575Z caller=node_exporter.go:117 level=info collector=uname 2024-03-22T05:41:50.655 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:41:50 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[62847]: ts=2024-03-22T05:41:50.575Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-22T05:41:50.655 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:41:50 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[62847]: ts=2024-03-22T05:41:50.575Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-22T05:41:50.655 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:41:50 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[62847]: ts=2024-03-22T05:41:50.575Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-22T05:41:50.655 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:41:50 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[62847]: ts=2024-03-22T05:41:50.575Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-22T05:41:50.655 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:41:50 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b[62847]: ts=2024-03-22T05:41:50.575Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-22T05:41:50.655 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:41:50 smithi186 bash[62797]: d7e0b3d201918d57c0675d79f678adb3a4c4d5a8cc2059bf7cdffb53411aa321 2024-03-22T05:41:50.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:50 smithi005 ceph-mon[24793]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:41:50.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:50 smithi005 ceph-mon[29523]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:41:51.043 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:41:50 smithi186 systemd[1]: Started Ceph node-exporter.b for f59f36e2-e80a-11ee-95cd-87774f69a715. 2024-03-22T05:41:51.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:51 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:51.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:51 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:51.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:51 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:41:51.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:51 smithi186 ceph-mon[25720]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:41:51.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:51 smithi186 ceph-mon[25720]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:41:51.819 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:51 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:51.819 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:51 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:51.819 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:51 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:41:51.819 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:51 smithi005 ceph-mon[29523]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:41:51.819 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:51 smithi005 ceph-mon[29523]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:41:51.819 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:51 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:51.819 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:51 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:51.819 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:51 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:41:51.819 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:51 smithi005 ceph-mon[24793]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:41:51.819 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:51 smithi005 ceph-mon[24793]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:41:53.135 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:52 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:41:53.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:52 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:41:53.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:52 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:41:54.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:54 smithi005 ceph-mon[24793]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:41:54.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:54 smithi005 ceph-mon[29523]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:41:54.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:54 smithi186 ceph-mon[25720]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:41:55.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:55 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:55.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:55 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:55.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:55 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:55.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:55 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:55.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:55 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:55.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:55 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:56.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:56 smithi186 ceph-mon[25720]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:41:56.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:56 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:56.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:56 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:56.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:56 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:56.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:56 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:56.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:56 smithi005 ceph-mon[24793]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:41:56.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:56 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:56.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:56 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:56.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:56 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:56.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:56 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:56.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:56 smithi005 ceph-mon[29523]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:41:56.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:56 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:56.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:56 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:56.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:56 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:56.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:56 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:57.452 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:41:56 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:41:56] "GET /metrics HTTP/1.1" 200 37979 "" "Prometheus/2.33.4" 2024-03-22T05:41:58.075 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:57 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:58.076 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:57 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:58.076 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:41:57 smithi005 ceph-mon[29523]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:41:58.076 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:57 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:58.076 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:57 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:58.076 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:41:57 smithi005 ceph-mon[24793]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:41:58.102 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:57 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:58.102 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:57 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:41:58.102 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:41:57 smithi186 ceph-mon[25720]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:42:00.243 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:00 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:00.243 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:00 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:00.243 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:00 smithi005 ceph-mon[29523]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:42:00.243 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:00 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:00.244 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:00 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:00.244 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:00 smithi005 ceph-mon[24793]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:42:00.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:00 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:00.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:00 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:00.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:00 smithi186 ceph-mon[25720]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:42:01.355 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:01 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:01.355 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:01 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:01.355 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:01 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:42:01.355 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:01 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:42:01.355 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:01 smithi005 ceph-mon[29523]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 937 B/s rd, 0 op/s 2024-03-22T05:42:01.355 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:01 smithi005 ceph-mon[29523]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:42:01.356 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:01 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:01.356 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:01 smithi005 ceph-mon[29523]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:42:01.356 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:01 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:42:01.356 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:01 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:42:01.356 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:01 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:42:01.356 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:01 smithi005 ceph-mon[29523]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:42:01.356 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:01 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:01.356 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:01 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:01.356 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:01 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:42:01.357 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:01 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:42:01.357 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:01 smithi005 ceph-mon[24793]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 937 B/s rd, 0 op/s 2024-03-22T05:42:01.357 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:01 smithi005 ceph-mon[24793]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:42:01.357 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:01 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:01.357 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:01 smithi005 ceph-mon[24793]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:42:01.357 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:01 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:42:01.357 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:01 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:42:01.357 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:01 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:42:01.357 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:01 smithi005 ceph-mon[24793]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:42:01.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:01 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:01.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:01 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:01.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:01 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:42:01.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:01 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:42:01.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:01 smithi186 ceph-mon[25720]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 937 B/s rd, 0 op/s 2024-03-22T05:42:01.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:01 smithi186 ceph-mon[25720]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:42:01.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:01 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:01.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:01 smithi186 ceph-mon[25720]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:42:01.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:01 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:42:01.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:01 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:42:01.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:01 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:42:01.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:01 smithi186 ceph-mon[25720]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:42:02.288 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:42:02.097+0000 7f487c563700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:42:02.289 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:42:02.289 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:42:02.289 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:42:02.289 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:42:02.289 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:02.289 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:02.289 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:42:02.289 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:42:02.289 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:42:02.289 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:42:02.289 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:42:02.289 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:42:02.289 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:42:02.290 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:42:02.290 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:42:02.290 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:42:02.290 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:42:02.290 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:42:02.290 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:02.290 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:02.291 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:42:02.291 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:42:02.291 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:42:02.291 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:42:02.291 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:42:02.291 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:42:02.291 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:42:02.291 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:42:02.291 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return future.result(timeout) 2024-03-22T05:42:02.291 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:42:02.291 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.__get_result() 2024-03-22T05:42:02.291 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:42:02.291 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: raise self._exception 2024-03-22T05:42:02.291 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:42:02.291 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: config_blobs=daemon_spec.final_config, 2024-03-22T05:42:02.292 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:42:02.292 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:42:02.292 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:42:02.292 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:42:02.292 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:42:02.292 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:42:02.292 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:42:02.292 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:02.292 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:02.292 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:42:02.292 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:42:02.292 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:42:02.292 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:42:02.292 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:42:02.293 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:42:02.293 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:42:02.293 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:42:02.293 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:42:02.293 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:42:02.293 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:42:02.293 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:42:02.293 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:02.293 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:02.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:02 smithi186 ceph-mon[25720]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:42:02.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:02 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:42:02.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:02 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:02.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:02 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:02.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:02 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:02.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:02 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:02.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:02 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:02.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:02 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:02.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:02 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:02.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:02 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:02.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:02 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:02.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:02 smithi005 ceph-mon[29523]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:42:02.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:02 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:42:02.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:02 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:02.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:02 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:02.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:02 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:02.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:02 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:02.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:02 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:02.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:02 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:02.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:02 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:02.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:02 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:02.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:02 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:02.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:02 smithi005 ceph-mon[24793]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:42:02.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:02 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:42:02.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:02 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:02.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:02 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:02.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:02 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:02.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:02 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:02.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:02 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:02.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:02 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:02.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:02 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:02.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:02 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:02.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:02 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:03.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:03 smithi005 ceph-mon[24793]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:42:03.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:42:03.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:42:03.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:42:03.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:42:03.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:03.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:03.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:42:03.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:42:03.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:42:03.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:42:03.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:42:03.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:42:03.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:42:03.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:42:03.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:42:03.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:42:03.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:42:03.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:42:03.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:03.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:03.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:42:03.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:42:03.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:42:03.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:42:03.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:42:03.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:42:03.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:42:03.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:42:03.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: return future.result(timeout) 2024-03-22T05:42:03.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:42:03.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.__get_result() 2024-03-22T05:42:03.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:42:03.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: raise self._exception 2024-03-22T05:42:03.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:42:03.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:42:03.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:42:03.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:42:03.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:42:03.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:42:03.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:42:03.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:42:03.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:42:03.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:03.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:03.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:42:03.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:42:03.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:42:03.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:42:03.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:42:03.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:42:03.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:42:03.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:42:03.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:42:03.707 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:03 smithi005 ceph-mon[29523]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:42:03.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:42:03.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:42:03.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:42:03.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:42:03.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:03.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:03.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:42:03.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:42:03.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:42:03.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:42:03.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:42:03.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:42:03.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:42:03.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:42:03.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:42:03.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:42:03.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:42:03.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:42:03.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:03.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:03.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:42:03.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:42:03.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:42:03.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:42:03.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:42:03.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:42:03.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:42:03.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:42:03.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: return future.result(timeout) 2024-03-22T05:42:03.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:42:03.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.__get_result() 2024-03-22T05:42:03.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:42:03.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: raise self._exception 2024-03-22T05:42:03.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:42:03.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:42:03.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:42:03.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:42:03.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:42:03.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:42:03.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:42:03.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:42:03.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:42:03.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:03.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:03.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:42:03.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:42:03.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:42:03.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:42:03.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:42:03.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:42:03.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:42:03.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:42:03.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:42:03.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:42:03.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:42:03.711 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:42:03.711 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:03.711 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:03.711 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:03 smithi005 ceph-mon[29523]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:42:03.711 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:42:03.711 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:42:03.711 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:42:03.711 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:03.711 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:03.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:03 smithi005 ceph-mon[24793]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:42:03.712 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:03 smithi186 ceph-mon[25720]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:42:03.712 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:42:03.712 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:42:03.712 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:42:03.712 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:42:03.712 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:03.712 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:03.712 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:42:03.712 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:42:03.713 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:42:03.713 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:42:03.713 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:42:03.713 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:42:03.713 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:42:03.713 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:42:03.713 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:42:03.713 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:42:03.713 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:42:03.713 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:42:03.713 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:03.713 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:03.713 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:42:03.713 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:42:03.713 INFO:journalctl@ceph.mon.b.smithi186.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:42:03.714 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:42:03.714 INFO:journalctl@ceph.mon.b.smithi186.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:42:03.714 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:42:03.714 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:42:03.714 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:42:03.714 INFO:journalctl@ceph.mon.b.smithi186.stdout: return future.result(timeout) 2024-03-22T05:42:03.714 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:42:03.714 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.__get_result() 2024-03-22T05:42:03.714 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:42:03.714 INFO:journalctl@ceph.mon.b.smithi186.stdout: raise self._exception 2024-03-22T05:42:03.714 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:42:03.714 INFO:journalctl@ceph.mon.b.smithi186.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:42:03.714 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:42:03.714 INFO:journalctl@ceph.mon.b.smithi186.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:42:03.714 INFO:journalctl@ceph.mon.b.smithi186.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:42:03.715 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:42:03.715 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:42:03.715 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:42:03.715 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:42:03.715 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:03.715 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:03.715 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:42:03.715 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:42:03.715 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:42:03.715 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:42:03.715 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:42:03.715 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:42:03.715 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:42:03.715 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:42:03.715 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:42:03.716 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:42:03.716 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:42:03.716 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:42:03.716 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:03.716 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:03.716 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:03 smithi186 ceph-mon[25720]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:42:05.612 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:05 smithi186 ceph-mon[25720]: Upgrade: Updating prometheus.a 2024-03-22T05:42:05.612 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:05 smithi186 ceph-mon[25720]: Deploying daemon prometheus.a on smithi186 2024-03-22T05:42:05.612 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:05 smithi186 ceph-mon[25720]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:42:05.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:05 smithi005 ceph-mon[24793]: Upgrade: Updating prometheus.a 2024-03-22T05:42:05.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:05 smithi005 ceph-mon[24793]: Deploying daemon prometheus.a on smithi186 2024-03-22T05:42:05.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:05 smithi005 ceph-mon[24793]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:42:05.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:05 smithi005 ceph-mon[29523]: Upgrade: Updating prometheus.a 2024-03-22T05:42:05.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:05 smithi005 ceph-mon[29523]: Deploying daemon prometheus.a on smithi186 2024-03-22T05:42:05.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:05 smithi005 ceph-mon[29523]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:42:07.401 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:42:06] "GET /metrics HTTP/1.1" 200 37979 "" "Prometheus/2.33.4" 2024-03-22T05:42:07.569 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:07 smithi186 ceph-mon[25720]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:42:07.570 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:42:07 smithi186 systemd[1]: Stopping Ceph prometheus.a for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:42:07.570 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:42:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[60807]: ts=2024-03-22T05:42:07.560Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-22T05:42:07.570 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:42:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[60807]: ts=2024-03-22T05:42:07.560Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-03-22T05:42:07.570 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:42:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[60807]: ts=2024-03-22T05:42:07.560Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-03-22T05:42:07.570 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:42:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[60807]: ts=2024-03-22T05:42:07.560Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-03-22T05:42:07.570 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:42:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[60807]: ts=2024-03-22T05:42:07.560Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-03-22T05:42:07.570 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:42:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[60807]: ts=2024-03-22T05:42:07.560Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-03-22T05:42:07.570 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:42:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[60807]: ts=2024-03-22T05:42:07.560Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-22T05:42:07.570 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:42:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[60807]: ts=2024-03-22T05:42:07.560Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-03-22T05:42:07.570 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:42:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[60807]: ts=2024-03-22T05:42:07.560Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-03-22T05:42:07.570 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:42:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[60807]: ts=2024-03-22T05:42:07.561Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-03-22T05:42:07.571 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:42:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[60807]: ts=2024-03-22T05:42:07.561Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-03-22T05:42:07.571 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:42:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[60807]: ts=2024-03-22T05:42:07.561Z caller=main.go:1066 level=info msg="See you next time!" 2024-03-22T05:42:07.571 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:42:07 smithi186 podman[64167]: 2024-03-22 05:42:07.57058144 +0000 UTC m=+0.111800976 container died 8596d431e5578a06c9264837e812c33e534a2dc1b1d0597ca291be9ba361cc05 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-22T05:42:07.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:07 smithi005 ceph-mon[24793]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:42:07.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:07 smithi005 ceph-mon[29523]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:42:08.243 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:42:07 smithi186 podman[64167]: 2024-03-22 05:42:07.976227509 +0000 UTC m=+0.517447044 container cleanup 8596d431e5578a06c9264837e812c33e534a2dc1b1d0597ca291be9ba361cc05 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-22T05:42:08.243 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:42:07 smithi186 bash[64167]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a 2024-03-22T05:42:08.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:08 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:42:08.543 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:42:08 smithi186 podman[64181]: 2024-03-22 05:42:08.244317398 +0000 UTC m=+0.675191677 container remove 8596d431e5578a06c9264837e812c33e534a2dc1b1d0597ca291be9ba361cc05 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-22T05:42:08.544 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:42:08 smithi186 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-03-22T05:42:08.544 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:42:08 smithi186 systemd[1]: Stopped Ceph prometheus.a for f59f36e2-e80a-11ee-95cd-87774f69a715. 2024-03-22T05:42:08.544 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:42:08 smithi186 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@prometheus.a.service: Consumed 1.279s CPU time. 2024-03-22T05:42:08.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:08 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:42:08.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:08 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:42:08.949 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:42:08 smithi186 systemd[1]: Starting Ceph prometheus.a for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:42:08.949 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:42:08 smithi186 podman[64313]: 2024-03-22 05:42:08.710138488 +0000 UTC m=+0.019103013 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-03-22T05:42:08.949 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:42:08 smithi186 podman[64313]: 2024-03-22 05:42:08.83713395 +0000 UTC m=+0.146098479 container create a5a043588d6dee234004c0e728608ee28c6c52f74391f789223f2eb6244d7dc8 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-22T05:42:09.351 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:42:09 smithi186 podman[64313]: 2024-03-22 05:42:09.089111928 +0000 UTC m=+0.398076454 container init a5a043588d6dee234004c0e728608ee28c6c52f74391f789223f2eb6244d7dc8 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-22T05:42:09.351 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:42:09 smithi186 podman[64313]: 2024-03-22 05:42:09.091930103 +0000 UTC m=+0.400894639 container start a5a043588d6dee234004c0e728608ee28c6c52f74391f789223f2eb6244d7dc8 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-22T05:42:09.351 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:42:09 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[64327]: ts=2024-03-22T05:42:09.116Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-03-22T05:42:09.351 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:42:09 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[64327]: ts=2024-03-22T05:42:09.116Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-03-22T05:42:09.351 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:42:09 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[64327]: ts=2024-03-22T05:42:09.116Z caller=main.go:570 level=info host_details="(Linux 5.14.0-430.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024 x86_64 smithi186 (none))" 2024-03-22T05:42:09.351 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:42:09 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[64327]: ts=2024-03-22T05:42:09.116Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-22T05:42:09.351 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:42:09 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[64327]: ts=2024-03-22T05:42:09.116Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-22T05:42:09.351 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:42:09 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[64327]: ts=2024-03-22T05:42:09.118Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-03-22T05:42:09.351 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:42:09 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[64327]: ts=2024-03-22T05:42:09.118Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-03-22T05:42:09.351 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:42:09 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[64327]: ts=2024-03-22T05:42:09.120Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-03-22T05:42:09.351 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:42:09 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[64327]: ts=2024-03-22T05:42:09.120Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-03-22T05:42:09.351 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:42:09 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[64327]: ts=2024-03-22T05:42:09.122Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-22T05:42:09.352 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:42:09 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[64327]: ts=2024-03-22T05:42:09.122Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.765µs 2024-03-22T05:42:09.352 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:42:09 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[64327]: ts=2024-03-22T05:42:09.122Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-22T05:42:09.352 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:42:09 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[64327]: ts=2024-03-22T05:42:09.129Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=6 2024-03-22T05:42:09.352 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:42:09 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[64327]: ts=2024-03-22T05:42:09.144Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=6 2024-03-22T05:42:09.352 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:42:09 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[64327]: ts=2024-03-22T05:42:09.152Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=6 2024-03-22T05:42:09.352 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:42:09 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[64327]: ts=2024-03-22T05:42:09.158Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=6 2024-03-22T05:42:09.352 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:42:09 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[64327]: ts=2024-03-22T05:42:09.161Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=4 maxSegment=6 2024-03-22T05:42:09.352 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:42:09 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[64327]: ts=2024-03-22T05:42:09.161Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=5 maxSegment=6 2024-03-22T05:42:09.352 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:42:09 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[64327]: ts=2024-03-22T05:42:09.162Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=6 maxSegment=6 2024-03-22T05:42:09.352 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:42:09 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[64327]: ts=2024-03-22T05:42:09.162Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=27.602µs wal_replay_duration=39.964273ms wbl_replay_duration=108ns total_replay_duration=40.009609ms 2024-03-22T05:42:09.352 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:42:09 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[64327]: ts=2024-03-22T05:42:09.165Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-03-22T05:42:09.352 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:42:09 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[64327]: ts=2024-03-22T05:42:09.165Z caller=main.go:1029 level=info msg="TSDB started" 2024-03-22T05:42:09.352 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:42:09 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[64327]: ts=2024-03-22T05:42:09.165Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-22T05:42:09.352 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:42:09 smithi186 bash[64313]: a5a043588d6dee234004c0e728608ee28c6c52f74391f789223f2eb6244d7dc8 2024-03-22T05:42:09.353 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:42:09 smithi186 systemd[1]: Started Ceph prometheus.a for f59f36e2-e80a-11ee-95cd-87774f69a715. 2024-03-22T05:42:09.353 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:42:09 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[64327]: ts=2024-03-22T05:42:09.186Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=20.828865ms db_storage=898ns remote_storage=1.261µs web_handler=421ns query_engine=595ns scrape=477.011µs scrape_sd=93.959µs notify=11.027µs notify_sd=14.506µs rules=19.92207ms tracing=7.778µs 2024-03-22T05:42:09.353 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:42:09 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[64327]: ts=2024-03-22T05:42:09.186Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-03-22T05:42:09.353 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:42:09 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[64327]: ts=2024-03-22T05:42:09.186Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-03-22T05:42:09.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:09 smithi005 ceph-mon[24793]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 573 B/s rd, 0 op/s 2024-03-22T05:42:09.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:09 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:09.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:09 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:09.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:09 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:42:09.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:09 smithi005 ceph-mon[29523]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 573 B/s rd, 0 op/s 2024-03-22T05:42:09.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:09 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:09.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:09 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:09.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:09 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:42:09.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:09 smithi186 ceph-mon[25720]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 573 B/s rd, 0 op/s 2024-03-22T05:42:09.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:09 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:09.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:09 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:09.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:09 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:42:11.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:11 smithi005 ceph-mon[24793]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:42:11.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:11 smithi005 ceph-mon[29523]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:42:11.790 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:11 smithi186 ceph-mon[25720]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:42:12.089 INFO:teuthology.orchestra.run.smithi005.stdout:true 2024-03-22T05:42:12.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:12 smithi005 ceph-mon[29523]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:42:12.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:12 smithi005 ceph-mon[24793]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:42:12.776 INFO:teuthology.orchestra.run.smithi005.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T05:42:12.776 INFO:teuthology.orchestra.run.smithi005.stdout:alertmanager.a smithi005 *:9093,9094 running (13m) 17s ago 21m 22.9M - 0.23.0 ba2b418f427c fd7dc5dbe3e5 2024-03-22T05:42:12.776 INFO:teuthology.orchestra.run.smithi005.stdout:grafana.a smithi186 *:3000 error 18s ago 21m - - 2024-03-22T05:42:12.776 INFO:teuthology.orchestra.run.smithi005.stdout:iscsi.foo.smithi005.mcdwbp smithi005 unknown 17s ago 20m 58.9M - 3.5 e1d6a67b021e 5215e03bf81b 2024-03-22T05:42:12.776 INFO:teuthology.orchestra.run.smithi005.stdout:mgr.x smithi186 *:8443,9283,8765 running (46s) 18s ago 26m 448M - 19.0.0-2299-gea01217f aa461e7c5e7f c556421a73f3 2024-03-22T05:42:12.776 INFO:teuthology.orchestra.run.smithi005.stdout:mgr.y smithi005 *:8443,9283,8765 running (10m) 17s ago 27m 500M - 19.0.0-2299-gea01217f aa461e7c5e7f 62a813eebc0d 2024-03-22T05:42:12.776 INFO:teuthology.orchestra.run.smithi005.stdout:mon.a smithi005 running (27m) 17s ago 27m 77.8M 2048M 17.2.0 e1d6a67b021e 38e0bba4bb03 2024-03-22T05:42:12.776 INFO:teuthology.orchestra.run.smithi005.stdout:mon.b smithi186 running (26m) 18s ago 26m 54.6M 2048M 17.2.0 e1d6a67b021e 323f53f932f6 2024-03-22T05:42:12.776 INFO:teuthology.orchestra.run.smithi005.stdout:mon.c smithi005 running (26m) 17s ago 26m 57.3M 2048M 17.2.0 e1d6a67b021e 00ec24d38612 2024-03-22T05:42:12.776 INFO:teuthology.orchestra.run.smithi005.stdout:node-exporter.a smithi005 *:9100 running (29s) 17s ago 22m 8312k - 1.5.0 0da6a335fe13 f635cd72b9d3 2024-03-22T05:42:12.776 INFO:teuthology.orchestra.run.smithi005.stdout:node-exporter.b smithi186 *:9100 running (22s) 18s ago 22m 3489k - 1.5.0 0da6a335fe13 d7e0b3d20191 2024-03-22T05:42:12.776 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0 smithi005 running (25m) 17s ago 25m 53.5M 2993M 17.2.0 e1d6a67b021e ef5c7053dd37 2024-03-22T05:42:12.777 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1 smithi005 running (25m) 17s ago 25m 56.5M 2993M 17.2.0 e1d6a67b021e a1e7ee1017b6 2024-03-22T05:42:12.777 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2 smithi005 running (25m) 17s ago 25m 48.5M 2993M 17.2.0 e1d6a67b021e 462ea63ad0e5 2024-03-22T05:42:12.777 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3 smithi005 running (24m) 17s ago 24m 56.5M 2993M 17.2.0 e1d6a67b021e b25432a7239b 2024-03-22T05:42:12.777 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4 smithi186 running (24m) 18s ago 24m 56.1M 3248M 17.2.0 e1d6a67b021e cd963671f340 2024-03-22T05:42:12.777 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5 smithi186 running (23m) 18s ago 23m 51.6M 3248M 17.2.0 e1d6a67b021e f98c87f1a2cd 2024-03-22T05:42:12.777 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6 smithi186 running (23m) 18s ago 23m 51.7M 3248M 17.2.0 e1d6a67b021e 704ac49c68c4 2024-03-22T05:42:12.777 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7 smithi186 running (23m) 18s ago 23m 54.1M 3248M 17.2.0 e1d6a67b021e f415b90edf63 2024-03-22T05:42:12.777 INFO:teuthology.orchestra.run.smithi005.stdout:prometheus.a smithi186 *:9095 starting - - - - 2024-03-22T05:42:12.777 INFO:teuthology.orchestra.run.smithi005.stdout:rgw.foo.smithi005.exdhqu smithi005 *:8000 running (20m) 17s ago 20m 78.7M - 17.2.0 e1d6a67b021e 6bb27fe11553 2024-03-22T05:42:12.777 INFO:teuthology.orchestra.run.smithi005.stdout:rgw.foo.smithi186.mcgodk smithi186 *:8000 running (20m) 18s ago 20m 77.0M - 17.2.0 e1d6a67b021e 19b5c20a34f7 2024-03-22T05:42:12.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:12 smithi186 ceph-mon[25720]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:42:13.182 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-03-22T05:42:13.182 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": { 2024-03-22T05:42:13.182 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-03-22T05:42:13.182 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:42:13.182 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr": { 2024-03-22T05:42:13.182 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev)": 2 2024-03-22T05:42:13.182 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:42:13.182 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": { 2024-03-22T05:42:13.182 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-22T05:42:13.182 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:42:13.182 INFO:teuthology.orchestra.run.smithi005.stdout: "mds": {}, 2024-03-22T05:42:13.183 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw": { 2024-03-22T05:42:13.183 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-22T05:42:13.183 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:42:13.183 INFO:teuthology.orchestra.run.smithi005.stdout: "overall": { 2024-03-22T05:42:13.183 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-03-22T05:42:13.183 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev)": 2 2024-03-22T05:42:13.183 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-03-22T05:42:13.183 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-03-22T05:42:13.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:13 smithi005 ceph-mon[29523]: from='client.25393 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:42:13.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:13 smithi005 ceph-mon[29523]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:42:13.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:13 smithi005 ceph-mon[29523]: from='client.15498 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:42:13.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:13 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:13.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:13 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:13.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:13 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/1591198857' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:13.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:13 smithi005 ceph-mon[24793]: from='client.25393 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:42:13.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:13 smithi005 ceph-mon[24793]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:42:13.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:13 smithi005 ceph-mon[24793]: from='client.15498 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:42:13.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:13 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:13.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:13 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:13.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:13 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/1591198857' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:13.532 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-03-22T05:42:13.532 INFO:teuthology.orchestra.run.smithi005.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9", 2024-03-22T05:42:13.532 INFO:teuthology.orchestra.run.smithi005.stdout: "in_progress": true, 2024-03-22T05:42:13.532 INFO:teuthology.orchestra.run.smithi005.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-03-22T05:42:13.532 INFO:teuthology.orchestra.run.smithi005.stdout: "services_complete": [ 2024-03-22T05:42:13.532 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr" 2024-03-22T05:42:13.532 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-03-22T05:42:13.532 INFO:teuthology.orchestra.run.smithi005.stdout: "progress": "2/2 daemons upgraded", 2024-03-22T05:42:13.532 INFO:teuthology.orchestra.run.smithi005.stdout: "message": "Currently upgrading prometheus daemons", 2024-03-22T05:42:13.532 INFO:teuthology.orchestra.run.smithi005.stdout: "is_paused": false 2024-03-22T05:42:13.533 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-03-22T05:42:13.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:13 smithi186 ceph-mon[25720]: from='client.25393 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:42:13.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:13 smithi186 ceph-mon[25720]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:42:13.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:13 smithi186 ceph-mon[25720]: from='client.15498 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:42:13.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:13 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:13.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:13 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:13.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:13 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/1591198857' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:14.691 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:14 smithi186 ceph-mon[25720]: from='client.25405 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:42:14.691 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:14 smithi186 ceph-mon[25720]: from='client.25417 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:42:14.691 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:14 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:14.691 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:14 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:14.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:14 smithi005 ceph-mon[29523]: from='client.25405 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:42:14.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:14 smithi005 ceph-mon[29523]: from='client.25417 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:42:14.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:14 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:14.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:14 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:14.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:14 smithi005 ceph-mon[24793]: from='client.25405 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:42:14.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:14 smithi005 ceph-mon[24793]: from='client.25417 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:42:14.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:14 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:14.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:14 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:15.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:15 smithi186 ceph-mon[25720]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:42:15.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:15 smithi005 ceph-mon[29523]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:42:15.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:15 smithi005 ceph-mon[24793]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:42:17.421 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:16 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:42:16] "GET /metrics HTTP/1.1" 200 37976 "" "Prometheus/2.43.0" 2024-03-22T05:42:17.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:17 smithi005 ceph-mon[29523]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:42:17.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:17 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:17.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:17 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:17.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:17 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:42:17.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:17 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:42:17.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:17 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:17.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:17 smithi005 ceph-mon[24793]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:42:17.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:17 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:17.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:17 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:17.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:17 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:42:17.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:17 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:42:17.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:17 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:17.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:17 smithi186 ceph-mon[25720]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:42:17.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:17 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:17.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:17 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:17.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:17 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:42:17.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:17 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:42:17.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:17 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:18.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:18 smithi005 ceph-mon[24793]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 924 B/s rd, 0 op/s 2024-03-22T05:42:18.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:18 smithi005 ceph-mon[24793]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:42:18.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:18 smithi005 ceph-mon[24793]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:42:18.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:18 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:42:18.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:18 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:42:18.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:18 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:42:18.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:18 smithi005 ceph-mon[24793]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:42:18.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:18 smithi005 ceph-mon[29523]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 924 B/s rd, 0 op/s 2024-03-22T05:42:18.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:18 smithi005 ceph-mon[29523]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:42:18.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:18 smithi005 ceph-mon[29523]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:42:18.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:18 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:42:18.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:18 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:42:18.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:18 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:42:18.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:18 smithi005 ceph-mon[29523]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:42:18.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:18 smithi186 ceph-mon[25720]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 924 B/s rd, 0 op/s 2024-03-22T05:42:18.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:18 smithi186 ceph-mon[25720]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:42:18.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:18 smithi186 ceph-mon[25720]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:42:18.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:18 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:42:18.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:18 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:42:18.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:18 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:42:18.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:18 smithi186 ceph-mon[25720]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:42:19.427 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:42:19.232+0000 7f487c563700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:42:19.428 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:42:19.428 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:42:19.428 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:42:19.428 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:42:19.428 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:19.429 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:19.429 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:42:19.429 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:42:19.429 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:42:19.429 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:42:19.429 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:42:19.429 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:42:19.429 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:42:19.429 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:42:19.430 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:42:19.430 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:42:19.430 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:42:19.430 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:42:19.431 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:19 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-22T05:42:19.431 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:19 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:42:19.431 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:19 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:19.431 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:19 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:19.431 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:19 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:19.431 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:19 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:19.431 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:19 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:19.431 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:19 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:19.432 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:19 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:19.432 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:19 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:19.432 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:19 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:19.432 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:19 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:19.432 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:19.433 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:19.433 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:42:19.433 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:42:19.433 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:42:19.433 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:42:19.433 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:42:19.433 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:42:19.433 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:42:19.433 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:42:19.433 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return future.result(timeout) 2024-03-22T05:42:19.433 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:42:19.433 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.__get_result() 2024-03-22T05:42:19.434 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:42:19.434 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: raise self._exception 2024-03-22T05:42:19.434 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:42:19.434 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: config_blobs=daemon_spec.final_config, 2024-03-22T05:42:19.434 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:42:19.434 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:42:19.434 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:42:19.434 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:42:19.434 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:42:19.434 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:42:19.435 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:42:19.435 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:19.435 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:19.435 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:42:19.435 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:42:19.435 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:42:19.435 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:42:19.435 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:42:19.435 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:42:19.435 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:42:19.435 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:42:19.435 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:42:19.436 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:42:19.436 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:42:19.436 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:42:19.436 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:19.436 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:19.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:19 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-22T05:42:19.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:19 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:42:19.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:19 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:19.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:19 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:19.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:19 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:19.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:19 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:19.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:19 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:19.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:19 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:19.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:19 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:19.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:19 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:19.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:19 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:19.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:19 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:19.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:19 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-22T05:42:19.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:19 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:42:19.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:19 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:19.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:19 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:19.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:19 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:19.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:19 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:19.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:19 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:19.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:19 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:19.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:19 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:19.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:19 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:19.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:19 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:19.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:19 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:20.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:20 smithi005 ceph-mon[24793]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:42:20.702 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:42:20.702 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:42:20.702 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:42:20.702 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:42:20.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:20.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:20.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:42:20.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:42:20.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:42:20.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:42:20.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:42:20.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:42:20.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:42:20.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:42:20.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:42:20.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:42:20.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:42:20.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:42:20.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:20.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:20.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:42:20.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:42:20.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:42:20.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:42:20.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:42:20.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:42:20.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:42:20.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:42:20.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: return future.result(timeout) 2024-03-22T05:42:20.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:42:20.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.__get_result() 2024-03-22T05:42:20.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:42:20.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: raise self._exception 2024-03-22T05:42:20.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:42:20.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:42:20.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:42:20.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:42:20.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:42:20.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:42:20.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:42:20.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:42:20.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:42:20.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:20.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:20.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:42:20.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:42:20.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:42:20.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:42:20.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:42:20.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:42:20.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:42:20.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:42:20.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:42:20.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:42:20.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:42:20.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:42:20.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:20.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:20.706 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:20 smithi005 ceph-mon[24793]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-22T05:42:20.706 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:20 smithi005 ceph-mon[24793]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:42:20.707 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:20 smithi005 ceph-mon[29523]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:42:20.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:42:20.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:42:20.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:42:20.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:42:20.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:20.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:20.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:42:20.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:42:20.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:42:20.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:42:20.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:42:20.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:42:20.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:42:20.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:42:20.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:42:20.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:42:20.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:42:20.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:42:20.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:20.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:20.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:42:20.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:42:20.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:42:20.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:42:20.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:42:20.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:42:20.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:42:20.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:42:20.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: return future.result(timeout) 2024-03-22T05:42:20.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:42:20.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.__get_result() 2024-03-22T05:42:20.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:42:20.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: raise self._exception 2024-03-22T05:42:20.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:42:20.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:42:20.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:42:20.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:42:20.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:42:20.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:42:20.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:42:20.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:42:20.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:42:20.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:20.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:20.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:42:20.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:42:20.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:42:20.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:42:20.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:42:20.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:42:20.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:42:20.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:42:20.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:42:20.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:42:20.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:42:20.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:42:20.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:20.711 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:20.711 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:20 smithi005 ceph-mon[29523]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-22T05:42:20.711 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:20 smithi005 ceph-mon[29523]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:42:20.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:20 smithi186 ceph-mon[25720]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:42:20.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:42:20.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:42:20.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:42:20.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:42:20.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:20.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:20.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:42:20.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:42:20.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:42:20.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:42:20.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:42:20.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:42:20.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:42:20.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:42:20.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:42:20.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:42:20.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:42:20.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:42:20.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:20.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:20.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:42:20.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:42:20.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:42:20.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:42:20.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:42:20.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:42:20.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:42:20.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:42:20.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: return future.result(timeout) 2024-03-22T05:42:20.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:42:20.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.__get_result() 2024-03-22T05:42:20.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:42:20.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: raise self._exception 2024-03-22T05:42:20.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:42:20.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:42:20.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:42:20.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:42:20.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:42:20.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:42:20.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:42:20.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:42:20.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:42:20.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:20.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:20.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:42:20.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:42:20.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:42:20.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:42:20.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:42:20.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:42:20.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:42:20.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:42:20.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:42:20.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:42:20.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:42:20.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:42:20.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:20.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:20.798 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:20 smithi186 ceph-mon[25720]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-22T05:42:20.798 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:20 smithi186 ceph-mon[25720]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:42:22.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:22 smithi005 ceph-mon[29523]: Upgrade: Updating alertmanager.a 2024-03-22T05:42:22.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:22 smithi005 ceph-mon[29523]: Deploying daemon alertmanager.a on smithi005 2024-03-22T05:42:22.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:22 smithi005 ceph-mon[29523]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:42:22.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:22 smithi005 ceph-mon[29523]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:42:22.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:22 smithi005 ceph-mon[24793]: Upgrade: Updating alertmanager.a 2024-03-22T05:42:22.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:22 smithi005 ceph-mon[24793]: Deploying daemon alertmanager.a on smithi005 2024-03-22T05:42:22.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:22 smithi005 ceph-mon[24793]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:42:22.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:22 smithi005 ceph-mon[24793]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:42:22.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:22 smithi186 ceph-mon[25720]: Upgrade: Updating alertmanager.a 2024-03-22T05:42:22.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:22 smithi186 ceph-mon[25720]: Deploying daemon alertmanager.a on smithi005 2024-03-22T05:42:22.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:22 smithi186 ceph-mon[25720]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:42:22.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:22 smithi186 ceph-mon[25720]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:42:23.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:23 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:42:23.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:23 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:42:23.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:23 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:42:24.702 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:42:24 smithi005 systemd[1]: Stopping Ceph alertmanager.a for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:42:24.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:24 smithi005 ceph-mon[24793]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 563 B/s rd, 0 op/s 2024-03-22T05:42:24.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:24 smithi005 ceph-mon[29523]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 563 B/s rd, 0 op/s 2024-03-22T05:42:24.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:24 smithi186 ceph-mon[25720]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 563 B/s rd, 0 op/s 2024-03-22T05:42:25.048 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:42:24 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[59009]: level=info ts=2024-03-22T05:42:24.705Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-03-22T05:42:25.048 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:42:24 smithi005 podman[78158]: 2024-03-22 05:42:24.746163368 +0000 UTC m=+0.162946441 container died fd7dc5dbe3e5ea6c4393721c9ae920cc999f81322c3cb7c4627af489527ce955 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-22T05:42:25.406 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:42:25 smithi005 podman[78158]: 2024-03-22 05:42:25.155859371 +0000 UTC m=+0.572642491 container cleanup fd7dc5dbe3e5ea6c4393721c9ae920cc999f81322c3cb7c4627af489527ce955 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-22T05:42:25.406 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:42:25 smithi005 bash[78158]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a 2024-03-22T05:42:25.702 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:42:25 smithi005 podman[78172]: 2024-03-22 05:42:25.406989189 +0000 UTC m=+0.653378729 container remove fd7dc5dbe3e5ea6c4393721c9ae920cc999f81322c3cb7c4627af489527ce955 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-22T05:42:25.702 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:42:25 smithi005 podman[78172]: 2024-03-22 05:42:25.491306417 +0000 UTC m=+0.737695966 volume remove bc04672c1062bdfcbaacb03f5b5fedd31d84be669ecff6a6528d2edd4d6e5d1b 2024-03-22T05:42:25.702 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:42:25 smithi005 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-22T05:42:25.702 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:42:25 smithi005 systemd[1]: Stopped Ceph alertmanager.a for f59f36e2-e80a-11ee-95cd-87774f69a715. 2024-03-22T05:42:25.702 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:42:25 smithi005 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@alertmanager.a.service: Consumed 1.524s CPU time. 2024-03-22T05:42:26.141 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:42:25 smithi005 systemd[1]: Starting Ceph alertmanager.a for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:42:26.141 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:42:26 smithi005 podman[78301]: 2024-03-22 05:42:26.033492277 +0000 UTC m=+0.113878904 volume create c10189d75c8a6b7410728eaaea072e6f92a48ffb533d17bd5fbc413f813faf77 2024-03-22T05:42:26.141 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:42:26 smithi005 podman[78301]: 2024-03-22 05:42:25.937394954 +0000 UTC m=+0.017781595 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-03-22T05:42:26.141 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:42:26 smithi005 podman[78301]: 2024-03-22 05:42:26.142280544 +0000 UTC m=+0.222667180 container create ee99dfc435e88501861d7a0d993f48ebb4ba59cba6956e9c8c82783e92ddad8e (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-22T05:42:26.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:26 smithi005 ceph-mon[24793]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:42:26.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:26 smithi005 ceph-mon[29523]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:42:26.703 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:42:26 smithi005 podman[78301]: 2024-03-22 05:42:26.428385258 +0000 UTC m=+0.508771900 container init ee99dfc435e88501861d7a0d993f48ebb4ba59cba6956e9c8c82783e92ddad8e (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-22T05:42:26.703 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:42:26 smithi005 podman[78301]: 2024-03-22 05:42:26.431117932 +0000 UTC m=+0.511504568 container start ee99dfc435e88501861d7a0d993f48ebb4ba59cba6956e9c8c82783e92ddad8e (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-22T05:42:26.703 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:42:26 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[78315]: ts=2024-03-22T05:42:26.443Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-03-22T05:42:26.703 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:42:26 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[78315]: ts=2024-03-22T05:42:26.443Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-03-22T05:42:26.703 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:42:26 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[78315]: ts=2024-03-22T05:42:26.445Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.5 port=9094 2024-03-22T05:42:26.703 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:42:26 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[78315]: ts=2024-03-22T05:42:26.447Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-22T05:42:26.703 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:42:26 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[78315]: ts=2024-03-22T05:42:26.470Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-22T05:42:26.703 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:42:26 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[78315]: ts=2024-03-22T05:42:26.470Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-22T05:42:26.703 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:42:26 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[78315]: ts=2024-03-22T05:42:26.473Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-03-22T05:42:26.703 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:42:26 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[78315]: ts=2024-03-22T05:42:26.473Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-03-22T05:42:26.703 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:42:26 smithi005 bash[78301]: ee99dfc435e88501861d7a0d993f48ebb4ba59cba6956e9c8c82783e92ddad8e 2024-03-22T05:42:26.703 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:42:26 smithi005 systemd[1]: Started Ceph alertmanager.a for f59f36e2-e80a-11ee-95cd-87774f69a715. 2024-03-22T05:42:26.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:26 smithi186 ceph-mon[25720]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:42:27.408 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:26 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:42:26] "GET /metrics HTTP/1.1" 200 37976 "" "Prometheus/2.43.0" 2024-03-22T05:42:27.665 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:27 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:27.666 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:27 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:27.666 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:27 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:42:27.666 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:27 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:27.666 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:27 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:27.666 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:27 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:42:27.695 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:27 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:27.695 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:27 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:27.695 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:27 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:42:28.702 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:42:28 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[78315]: ts=2024-03-22T05:42:28.447Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.00042882s 2024-03-22T05:42:28.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:28 smithi005 ceph-mon[24793]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:42:28.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:28 smithi005 ceph-mon[29523]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:42:28.911 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:28 smithi186 ceph-mon[25720]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:42:30.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:30 smithi005 ceph-mon[24793]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:42:30.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:30 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:30.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:30 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:30.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:30 smithi005 ceph-mon[29523]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:42:30.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:30 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:30.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:30 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:31.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:30 smithi186 ceph-mon[25720]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:42:31.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:30 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:31.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:30 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:32.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:31 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:32.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:31 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:32.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:31 smithi186 ceph-mon[25720]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:42:32.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:31 smithi186 ceph-mon[25720]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:42:32.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:31 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:32.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:31 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:32.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:31 smithi005 ceph-mon[24793]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:42:32.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:31 smithi005 ceph-mon[24793]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:42:32.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:31 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:32.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:31 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:32.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:31 smithi005 ceph-mon[29523]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:42:32.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:31 smithi005 ceph-mon[29523]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:42:33.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:33 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:33.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:33 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:33.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:33 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:33.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:33 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:33.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:33 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:33.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:33 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:34.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:34 smithi005 ceph-mon[29523]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:42:34.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:34 smithi005 ceph-mon[24793]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:42:34.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:34 smithi186 ceph-mon[25720]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:42:36.702 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:42:36 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[78315]: ts=2024-03-22T05:42:36.450Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.00359885s 2024-03-22T05:42:36.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:36 smithi005 ceph-mon[24793]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:42:36.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:36 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:36.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:36 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:36.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:36 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:42:36.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:36 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:42:36.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:36 smithi005 ceph-mon[24793]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 994 B/s rd, 0 op/s 2024-03-22T05:42:36.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:36 smithi005 ceph-mon[24793]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:42:36.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:36 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:36.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:36 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:42:36.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:36 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:42:36.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:36 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:42:36.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:36 smithi005 ceph-mon[29523]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:42:36.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:36 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:36.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:36 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:36.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:36 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:42:36.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:36 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:42:36.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:36 smithi005 ceph-mon[29523]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 994 B/s rd, 0 op/s 2024-03-22T05:42:36.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:36 smithi005 ceph-mon[29523]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:42:36.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:36 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:36.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:36 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:42:36.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:36 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:42:36.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:36 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:42:37.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:36 smithi186 ceph-mon[25720]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:42:37.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:36 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:37.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:36 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:37.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:36 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:42:37.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:36 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:42:37.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:36 smithi186 ceph-mon[25720]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 994 B/s rd, 0 op/s 2024-03-22T05:42:37.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:36 smithi186 ceph-mon[25720]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:42:37.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:36 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:37.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:36 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:42:37.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:36 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:42:37.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:36 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:42:37.202 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:36 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:42:36] "GET /metrics HTTP/1.1" 200 37976 "" "Prometheus/2.43.0" 2024-03-22T05:42:37.674 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:42:37.484+0000 7f487c563700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:42:37.674 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:42:37.675 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:42:37.675 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:42:37.675 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:42:37.675 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:37.675 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:37.675 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:42:37.675 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:42:37.675 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:42:37.675 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:42:37.675 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:42:37.676 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:42:37.676 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:42:37.676 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:42:37.676 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:42:37.676 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:42:37.676 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:42:37.676 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:42:37.676 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:37.677 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:37.677 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:42:37.677 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:42:37.677 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:42:37.677 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:42:37.677 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:42:37.677 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:42:37.677 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:42:37.677 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:42:37.677 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return future.result(timeout) 2024-03-22T05:42:37.677 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:42:37.677 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.__get_result() 2024-03-22T05:42:37.677 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:42:37.677 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: raise self._exception 2024-03-22T05:42:37.678 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:42:37.678 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: config_blobs=daemon_spec.final_config, 2024-03-22T05:42:37.678 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:42:37.678 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:42:37.678 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:42:37.678 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:42:37.678 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:42:37.678 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:42:37.678 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:42:37.678 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:37.678 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:37.678 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:42:37.678 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:42:37.678 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:42:37.679 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:42:37.679 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:42:37.679 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:42:37.679 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:42:37.679 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:42:37.679 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:42:37.679 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:42:37.679 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:42:37.679 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:42:37.679 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:37.679 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:37.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-mon[24793]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:42:37.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-mon[24793]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:42:37.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-22T05:42:37.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:42:37.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:37.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:37.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:37.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:37.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:37.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:37.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:37.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:37.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:37.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:37.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:37.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-mon[29523]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:42:37.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-mon[29523]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:42:37.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-22T05:42:37.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:42:37.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:37.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:37.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:37.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:37.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:37.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:37.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:37.955 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:37.955 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:37.955 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:37.955 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:37 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:38.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:37 smithi186 ceph-mon[25720]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:42:38.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:37 smithi186 ceph-mon[25720]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:42:38.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:37 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-22T05:42:38.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:37 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:42:38.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:37 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:38.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:37 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:38.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:37 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:38.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:37 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:38.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:37 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:38.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:37 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:38.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:37 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:38.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:37 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:38.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:37 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:38.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:37 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:38.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:37 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:38.936 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:38 smithi005 ceph-mon[24793]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:42:38.936 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:42:38.936 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:42:38.936 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:42:38.936 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:42:38.937 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:38.937 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:38.937 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:42:38.937 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:42:38.937 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:42:38.937 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:42:38.937 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:42:38.937 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:42:38.937 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:42:38.938 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:42:38.938 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:42:38.938 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:42:38.938 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:42:38.938 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:42:38.938 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:38.938 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:38.938 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:42:38.938 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:42:38.938 INFO:journalctl@ceph.mon.a.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:42:38.938 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:42:38.938 INFO:journalctl@ceph.mon.a.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:42:38.938 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:42:38.939 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:42:38.939 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:42:38.939 INFO:journalctl@ceph.mon.a.smithi005.stdout: return future.result(timeout) 2024-03-22T05:42:38.939 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:42:38.939 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.__get_result() 2024-03-22T05:42:38.939 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:42:38.939 INFO:journalctl@ceph.mon.a.smithi005.stdout: raise self._exception 2024-03-22T05:42:38.939 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:42:38.939 INFO:journalctl@ceph.mon.a.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:42:38.939 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:42:38.939 INFO:journalctl@ceph.mon.a.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:42:38.939 INFO:journalctl@ceph.mon.a.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:42:38.939 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:42:38.939 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:42:38.940 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:42:38.940 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:42:38.940 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:38.940 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:38.940 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:42:38.940 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:42:38.940 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:42:38.940 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:42:38.940 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:42:38.940 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:42:38.940 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:42:38.940 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:42:38.940 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:42:38.940 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:42:38.940 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:42:38.941 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:42:38.941 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:38.941 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:38.941 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:38 smithi005 ceph-mon[24793]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-22T05:42:38.941 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:38 smithi005 ceph-mon[24793]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 616 B/s rd, 0 op/s 2024-03-22T05:42:38.941 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:38 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:42:38.941 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:38 smithi005 ceph-mon[29523]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:42:38.941 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:42:38.942 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:42:38.942 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:42:38.942 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:42:38.942 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:38.942 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:38.942 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:42:38.942 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:42:38.942 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:42:38.942 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:42:38.942 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:42:38.942 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:42:38.942 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:42:38.942 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:42:38.943 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:42:38.943 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:42:38.943 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:42:38.943 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:42:38.943 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:38.943 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:38.943 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:42:38.943 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:42:38.943 INFO:journalctl@ceph.mon.c.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:42:38.943 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:42:38.943 INFO:journalctl@ceph.mon.c.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:42:38.944 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:42:38.944 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:42:38.944 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:42:38.944 INFO:journalctl@ceph.mon.c.smithi005.stdout: return future.result(timeout) 2024-03-22T05:42:38.944 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:42:38.944 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.__get_result() 2024-03-22T05:42:38.944 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:42:38.944 INFO:journalctl@ceph.mon.c.smithi005.stdout: raise self._exception 2024-03-22T05:42:38.944 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:42:38.944 INFO:journalctl@ceph.mon.c.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:42:38.944 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:42:38.945 INFO:journalctl@ceph.mon.c.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:42:38.945 INFO:journalctl@ceph.mon.c.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:42:38.945 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:42:38.945 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:42:38.945 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:42:38.945 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:42:38.945 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:38.945 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:38.945 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:42:38.945 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:42:38.945 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:42:38.945 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:42:38.945 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:42:38.945 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:42:38.946 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:42:38.946 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:42:38.946 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:42:38.946 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:42:38.946 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:42:38.946 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:42:38.946 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:38.946 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:38.946 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:38 smithi005 ceph-mon[29523]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-22T05:42:38.946 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:38 smithi005 ceph-mon[29523]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 616 B/s rd, 0 op/s 2024-03-22T05:42:38.946 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:38 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:42:38.973 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:38 smithi186 ceph-mon[25720]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:42:38.973 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:42:38.974 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:42:38.974 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:42:38.974 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:42:38.974 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:38.974 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:38.974 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:42:38.974 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:42:38.974 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:42:38.974 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:42:38.975 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:42:38.975 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:42:38.975 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:42:38.975 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:42:38.975 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:42:38.975 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:42:38.975 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:42:38.975 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:42:38.975 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:38.975 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:38.975 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:42:38.975 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:42:38.976 INFO:journalctl@ceph.mon.b.smithi186.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:42:38.976 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:42:38.976 INFO:journalctl@ceph.mon.b.smithi186.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:42:38.976 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:42:38.976 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:42:38.976 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:42:38.976 INFO:journalctl@ceph.mon.b.smithi186.stdout: return future.result(timeout) 2024-03-22T05:42:38.976 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:42:38.976 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.__get_result() 2024-03-22T05:42:38.976 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:42:38.976 INFO:journalctl@ceph.mon.b.smithi186.stdout: raise self._exception 2024-03-22T05:42:38.976 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:42:38.976 INFO:journalctl@ceph.mon.b.smithi186.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:42:38.977 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:42:38.977 INFO:journalctl@ceph.mon.b.smithi186.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:42:38.977 INFO:journalctl@ceph.mon.b.smithi186.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:42:38.977 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:42:38.977 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:42:38.977 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:42:38.977 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:42:38.977 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:38.977 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:38.977 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:42:38.977 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:42:38.977 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:42:38.978 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:42:38.978 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:42:38.978 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:42:38.978 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:42:38.978 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:42:38.978 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:42:38.978 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:42:38.978 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:42:38.978 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:42:38.978 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:38.978 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:38.978 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:38 smithi186 ceph-mon[25720]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-22T05:42:38.979 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:38 smithi186 ceph-mon[25720]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 616 B/s rd, 0 op/s 2024-03-22T05:42:38.979 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:38 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:42:40.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:39 smithi186 ceph-mon[25720]: Upgrade: Updating grafana.a 2024-03-22T05:42:40.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:39 smithi186 ceph-mon[25720]: Deploying daemon grafana.a on smithi186 2024-03-22T05:42:40.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:39 smithi186 ceph-mon[25720]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:42:40.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:39 smithi005 ceph-mon[24793]: Upgrade: Updating grafana.a 2024-03-22T05:42:40.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:39 smithi005 ceph-mon[24793]: Deploying daemon grafana.a on smithi186 2024-03-22T05:42:40.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:39 smithi005 ceph-mon[24793]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:42:40.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:39 smithi005 ceph-mon[29523]: Upgrade: Updating grafana.a 2024-03-22T05:42:40.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:39 smithi005 ceph-mon[29523]: Deploying daemon grafana.a on smithi186 2024-03-22T05:42:40.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:39 smithi005 ceph-mon[29523]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:42:41.912 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:41 smithi186 ceph-mon[25720]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:42:41.912 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:41 smithi186 ceph-mon[25720]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:42:41.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:41 smithi005 ceph-mon[24793]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:42:41.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:41 smithi005 ceph-mon[24793]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:42:41.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:41 smithi005 ceph-mon[29523]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:42:41.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:41 smithi005 ceph-mon[29523]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:42:42.293 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 systemd[1]: Starting Ceph grafana.a for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:42:42.641 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 podman[66274]: 2024-03-22 05:42:42.21286556 +0000 UTC m=+0.018527558 image pull 2bacad6d85d802cd176c146528adcc0e47a8a83e94bc4a8b7ba1a9b9ef2c65e7 quay.io/ceph/grafana:9.4.12 2024-03-22T05:42:42.641 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 podman[66274]: 2024-03-22 05:42:42.323194065 +0000 UTC m=+0.128856055 container create 16fd57dbbd7124c4d8f0f49a30de3b13acc12f8f5c9887ebc108931fa1c6ecc6 (image=quay.io/ceph/grafana:9.4.12, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a, maintainer=Grafana Labs ) 2024-03-22T05:42:42.641 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 podman[66274]: 2024-03-22 05:42:42.550625882 +0000 UTC m=+0.356287894 container init 16fd57dbbd7124c4d8f0f49a30de3b13acc12f8f5c9887ebc108931fa1c6ecc6 (image=quay.io/ceph/grafana:9.4.12, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a, maintainer=Grafana Labs ) 2024-03-22T05:42:42.641 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 podman[66274]: 2024-03-22 05:42:42.553260943 +0000 UTC m=+0.358922935 container start 16fd57dbbd7124c4d8f0f49a30de3b13acc12f8f5c9887ebc108931fa1c6ecc6 (image=quay.io/ceph/grafana:9.4.12, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a, maintainer=Grafana Labs ) 2024-03-22T05:42:42.892 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 bash[66274]: 16fd57dbbd7124c4d8f0f49a30de3b13acc12f8f5c9887ebc108931fa1c6ecc6 2024-03-22T05:42:42.892 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 systemd[1]: Started Ceph grafana.a for f59f36e2-e80a-11ee-95cd-87774f69a715. 2024-03-22T05:42:42.892 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=settings t=2024-03-22T05:42:42.706954845Z level=info msg="Starting Grafana" version=9.4.12 commit=37ee25ba46 branch=HEAD compiled=2023-05-22T15:58:05Z 2024-03-22T05:42:42.892 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=settings t=2024-03-22T05:42:42.707127196Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-03-22T05:42:42.892 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=settings t=2024-03-22T05:42:42.707139118Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-03-22T05:42:42.892 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=settings t=2024-03-22T05:42:42.707142629Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-03-22T05:42:42.892 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=settings t=2024-03-22T05:42:42.707145373Z level=info msg="Config overridden from command line" arg="default.paths.data=/var/lib/grafana" 2024-03-22T05:42:42.892 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=settings t=2024-03-22T05:42:42.707148116Z level=info msg="Config overridden from command line" arg="default.paths.logs=/var/log/grafana" 2024-03-22T05:42:42.892 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=settings t=2024-03-22T05:42:42.707150777Z level=info msg="Config overridden from command line" arg="default.paths.plugins=/var/lib/grafana/plugins" 2024-03-22T05:42:42.892 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=settings t=2024-03-22T05:42:42.707153663Z level=info msg="Config overridden from command line" arg="default.paths.provisioning=/etc/grafana/provisioning" 2024-03-22T05:42:42.893 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=settings t=2024-03-22T05:42:42.707156461Z level=info msg="Config overridden from command line" arg="default.log.mode=console" 2024-03-22T05:42:42.893 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=settings t=2024-03-22T05:42:42.707160697Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-03-22T05:42:42.893 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=settings t=2024-03-22T05:42:42.707163608Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-03-22T05:42:42.893 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=settings t=2024-03-22T05:42:42.707166647Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-03-22T05:42:42.893 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=settings t=2024-03-22T05:42:42.707169168Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-03-22T05:42:42.893 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=settings t=2024-03-22T05:42:42.707172054Z level=info msg="Path Home" path=/usr/share/grafana 2024-03-22T05:42:42.893 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=settings t=2024-03-22T05:42:42.707174662Z level=info msg="Path Data" path=/var/lib/grafana 2024-03-22T05:42:42.893 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=settings t=2024-03-22T05:42:42.707177115Z level=info msg="Path Logs" path=/var/log/grafana 2024-03-22T05:42:42.893 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=settings t=2024-03-22T05:42:42.707179484Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-03-22T05:42:42.894 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=settings t=2024-03-22T05:42:42.707182128Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-03-22T05:42:42.894 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=settings t=2024-03-22T05:42:42.707184936Z level=info msg="App mode production" 2024-03-22T05:42:42.894 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=sqlstore t=2024-03-22T05:42:42.707233234Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-03-22T05:42:42.894 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=sqlstore t=2024-03-22T05:42:42.707244596Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-03-22T05:42:42.894 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.718381443Z level=info msg="Starting DB migrations" 2024-03-22T05:42:42.894 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.721622239Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-03-22T05:42:42.894 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.732822805Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-03-22T05:42:42.894 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.733940962Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-03-22T05:42:42.894 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.736775621Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-03-22T05:42:42.895 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.739497895Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-03-22T05:42:42.895 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.742064318Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-03-22T05:42:42.895 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.744801506Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-03-22T05:42:42.895 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.747536927Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-03-22T05:42:42.895 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.750361628Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-03-22T05:42:42.895 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.752954846Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-03-22T05:42:42.895 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.75375997Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-03-22T05:42:42.895 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.755109677Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-03-22T05:42:42.895 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.756229146Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-03-22T05:42:42.895 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.757366533Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-03-22T05:42:42.895 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.75847538Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-03-22T05:42:42.895 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.759544652Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-03-22T05:42:42.895 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.760406249Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-03-22T05:42:42.895 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.76147458Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-03-22T05:42:42.895 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.76254555Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-03-22T05:42:42.896 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.763407846Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-03-22T05:42:42.896 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.764477915Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-03-22T05:42:42.896 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.765620016Z level=info msg="Executing migration" id="create provenance_type table" 2024-03-22T05:42:42.896 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.766055682Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-03-22T05:42:42.896 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.76692013Z level=info msg="Executing migration" id="create alert_image table" 2024-03-22T05:42:42.896 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.767747125Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-03-22T05:42:42.896 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.768622436Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-03-22T05:42:42.896 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.768932283Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-03-22T05:42:42.896 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.769831429Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-03-22T05:42:42.896 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.770623063Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-03-22T05:42:42.896 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.770710292Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-03-22T05:42:42.896 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.771584266Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-03-22T05:42:42.896 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.772201288Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-03-22T05:42:42.896 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.77304524Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-03-22T05:42:42.896 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.773866002Z level=info msg="Executing migration" id="create secrets table" 2024-03-22T05:42:42.897 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.775422454Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-03-22T05:42:42.897 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.781667673Z level=info msg="Executing migration" id="add name column into data_keys" 2024-03-22T05:42:42.897 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.782793875Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-03-22T05:42:42.897 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.78367261Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-03-22T05:42:42.897 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.790056095Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-03-22T05:42:42.897 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.796478006Z level=info msg="Executing migration" id="add column hidden to role table" 2024-03-22T05:42:42.897 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.797612025Z level=info msg="Executing migration" id="create query_history table v1" 2024-03-22T05:42:42.897 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.798478396Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-03-22T05:42:42.897 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.799318848Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-03-22T05:42:42.897 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.799623079Z level=info msg="Executing migration" id="teams permissions migration" 2024-03-22T05:42:42.897 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.800473962Z level=info msg="Executing migration" id="dashboard permissions" 2024-03-22T05:42:42.897 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.801876529Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-03-22T05:42:42.897 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.804680616Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-03-22T05:42:42.897 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.807492165Z level=info msg="Executing migration" id="alerting notification permissions" 2024-03-22T05:42:42.897 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.810335465Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-03-22T05:42:42.898 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.813193417Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-03-22T05:42:42.898 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.81592816Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-03-22T05:42:42.898 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.81707293Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-03-22T05:42:42.898 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.81997792Z level=info msg="Executing migration" id="create correlation table v1" 2024-03-22T05:42:42.898 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.822733364Z level=info msg="Executing migration" id="add index correlations.uid" 2024-03-22T05:42:42.898 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.825626866Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-03-22T05:42:42.898 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.826071737Z level=info msg="Executing migration" id="add correlation config column" 2024-03-22T05:42:42.898 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.828882033Z level=info msg="Executing migration" id="create entity_events table" 2024-03-22T05:42:42.898 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.831633307Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-03-22T05:42:42.898 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.834501377Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-22T05:42:42.898 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.834593262Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-22T05:42:42.898 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.834868407Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-22T05:42:42.899 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.834948471Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-22T05:42:42.899 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.837680758Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-03-22T05:42:42.899 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.84055513Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-03-22T05:42:42.899 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.843355905Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-03-22T05:42:42.899 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.843899565Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-22T05:42:42.899 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.846561841Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-03-22T05:42:42.899 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.849401059Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-22T05:42:42.899 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.852162971Z level=info msg="Executing migration" id="Drop public config table" 2024-03-22T05:42:42.900 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.852598629Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-03-22T05:42:42.900 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.855301913Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-03-22T05:42:42.900 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.858047557Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-22T05:42:42.900 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.860782556Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-03-22T05:42:42.900 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.861243228Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-03-22T05:42:42.900 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.865149384Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-03-22T05:42:42.900 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.867872163Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-03-22T05:42:42.900 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.870613859Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-03-22T05:42:42.900 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.870950511Z level=info msg="Executing migration" id="add share column" 2024-03-22T05:42:42.900 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.873641238Z level=info msg="Executing migration" id="create default alerting folders" 2024-03-22T05:42:42.900 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.876360211Z level=info msg="Executing migration" id="create file table" 2024-03-22T05:42:42.901 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.879124693Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-03-22T05:42:42.901 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.879629929Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-03-22T05:42:42.901 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.882105419Z level=info msg="Executing migration" id="create file_meta table" 2024-03-22T05:42:42.901 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.882950068Z level=info msg="Executing migration" id="file table idx: path key" 2024-03-22T05:42:42.901 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.883764653Z level=info msg="Executing migration" id="set path collation in file table" 2024-03-22T05:42:42.901 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.884606581Z level=info msg="Executing migration" id="managed permissions migration" 2024-03-22T05:42:42.901 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.885769201Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-03-22T05:42:42.901 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.886609457Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-03-22T05:42:42.901 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.887465773Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-03-22T05:42:42.901 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.888696555Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-03-22T05:42:42.901 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.889488331Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-03-22T05:42:42.901 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.890322234Z level=info msg="Executing migration" id="update group index for alert rules" 2024-03-22T05:42:42.901 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.89116197Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-03-22T05:42:42.901 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.891731182Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-03-22T05:42:42.902 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.892581963Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-03-22T05:42:42.902 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.893914932Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-03-22T05:42:42.902 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.895165331Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-03-22T05:42:42.902 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.895670913Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-03-22T05:42:43.294 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.908939105Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-03-22T05:42:43.294 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.909765003Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-03-22T05:42:43.294 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.910614448Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-03-22T05:42:43.294 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.915039092Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-03-22T05:42:43.294 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.915872867Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-03-22T05:42:43.294 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.916719622Z level=info msg="Executing migration" id="create folder table" 2024-03-22T05:42:43.294 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.917564051Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-03-22T05:42:43.294 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.918028206Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-03-22T05:42:43.294 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.918931731Z level=info msg="Executing migration" id="Update folder title length" 2024-03-22T05:42:43.294 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.919784845Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-03-22T05:42:43.294 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=migrator t=2024-03-22T05:42:42.9206883Z level=info msg="migrations completed" performed=105 skipped=376 duration=199.09878ms 2024-03-22T05:42:43.294 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=sqlstore t=2024-03-22T05:42:42.920994931Z level=info msg="Created default organization" 2024-03-22T05:42:43.294 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=plugin.finder t=2024-03-22T05:42:42.954859969Z level=warn msg="Skipping finding plugins as directory does not exist" path=/usr/share/grafana/plugins-bundled 2024-03-22T05:42:43.295 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=secrets t=2024-03-22T05:42:42.955119172Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-03-22T05:42:43.295 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=query_data t=2024-03-22T05:42:42.957569677Z level=info msg="Query Service initialization" 2024-03-22T05:42:43.295 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:42 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=live.push_http t=2024-03-22T05:42:42.959993056Z level=info msg="Live Push Gateway initialization" 2024-03-22T05:42:43.896 INFO:teuthology.orchestra.run.smithi005.stdout:true 2024-03-22T05:42:43.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:43 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:43.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:43 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:43.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:43 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:42:43.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:43 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:43.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:43 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:43.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:43 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:42:44.046 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:43 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:44.046 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:43 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:44.046 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:43 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:42:44.046 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:43 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=infra.usagestats.collector t=2024-03-22T05:42:43.62147331Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-03-22T05:42:44.047 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:43 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=provisioning.datasources t=2024-03-22T05:42:43.623576012Z level=info msg="deleted correlations based on configuration" ds_name=Dashboard1 2024-03-22T05:42:44.047 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:43 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=provisioning.datasources t=2024-03-22T05:42:43.623585634Z level=info msg="deleted datasource based on configuration" name=Dashboard1 2024-03-22T05:42:44.047 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:43 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=provisioning.datasources t=2024-03-22T05:42:43.623680402Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-03-22T05:42:44.047 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:43 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=provisioning.datasources t=2024-03-22T05:42:43.634761177Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-03-22T05:42:44.047 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:43 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=provisioning.alerting t=2024-03-22T05:42:43.645500228Z level=info msg="starting to provision alerting" 2024-03-22T05:42:44.047 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:43 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=provisioning.alerting t=2024-03-22T05:42:43.645515698Z level=info msg="finished to provision alerting" 2024-03-22T05:42:44.047 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:43 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=grafanaStorageLogger t=2024-03-22T05:42:43.646121982Z level=info msg="storage starting" 2024-03-22T05:42:44.047 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:43 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=ngalert.state.manager t=2024-03-22T05:42:43.646934867Z level=info msg="Warming state cache for startup" 2024-03-22T05:42:44.047 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:43 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=ngalert.state.manager t=2024-03-22T05:42:43.647335941Z level=info msg="State cache has been initialized" states=0 duration=399.723µs 2024-03-22T05:42:44.047 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:43 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=ticker t=2024-03-22T05:42:43.647535579Z level=info msg=starting first_tick=2024-03-22T05:42:50Z 2024-03-22T05:42:44.047 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:43 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=ngalert.multiorg.alertmanager t=2024-03-22T05:42:43.648044847Z level=info msg="starting MultiOrg Alertmanager" 2024-03-22T05:42:44.047 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:43 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=http.server t=2024-03-22T05:42:43.653002935Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-03-22T05:42:44.047 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:43 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=sqlstore.transactions t=2024-03-22T05:42:43.664242792Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-22T05:42:44.048 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:43 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=sqlstore.transactions t=2024-03-22T05:42:43.675200344Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked" 2024-03-22T05:42:44.048 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:42:43 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=sqlstore.transactions t=2024-03-22T05:42:43.68664144Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=2 code="database is locked" 2024-03-22T05:42:44.564 INFO:teuthology.orchestra.run.smithi005.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T05:42:44.565 INFO:teuthology.orchestra.run.smithi005.stdout:alertmanager.a smithi005 *:9093,9094 running (18s) 13s ago 22m 17.1M - 0.25.0 c8568f914cd2 ee99dfc435e8 2024-03-22T05:42:44.565 INFO:teuthology.orchestra.run.smithi005.stdout:grafana.a smithi186 *:3000 starting - - - - 2024-03-22T05:42:44.565 INFO:teuthology.orchestra.run.smithi005.stdout:iscsi.foo.smithi005.mcdwbp smithi005 unknown 13s ago 21m 58.9M - 3.5 e1d6a67b021e 5215e03bf81b 2024-03-22T05:42:44.565 INFO:teuthology.orchestra.run.smithi005.stdout:mgr.x smithi186 *:8443,9283,8765 running (78s) 14s ago 26m 449M - 19.0.0-2299-gea01217f aa461e7c5e7f c556421a73f3 2024-03-22T05:42:44.565 INFO:teuthology.orchestra.run.smithi005.stdout:mgr.y smithi005 *:8443,9283,8765 running (11m) 13s ago 28m 503M - 19.0.0-2299-gea01217f aa461e7c5e7f 62a813eebc0d 2024-03-22T05:42:44.565 INFO:teuthology.orchestra.run.smithi005.stdout:mon.a smithi005 running (28m) 13s ago 28m 78.5M 2048M 17.2.0 e1d6a67b021e 38e0bba4bb03 2024-03-22T05:42:44.565 INFO:teuthology.orchestra.run.smithi005.stdout:mon.b smithi186 running (26m) 14s ago 26m 54.9M 2048M 17.2.0 e1d6a67b021e 323f53f932f6 2024-03-22T05:42:44.565 INFO:teuthology.orchestra.run.smithi005.stdout:mon.c smithi005 running (27m) 13s ago 27m 58.8M 2048M 17.2.0 e1d6a67b021e 00ec24d38612 2024-03-22T05:42:44.565 INFO:teuthology.orchestra.run.smithi005.stdout:node-exporter.a smithi005 *:9100 running (60s) 13s ago 22m 8786k - 1.5.0 0da6a335fe13 f635cd72b9d3 2024-03-22T05:42:44.565 INFO:teuthology.orchestra.run.smithi005.stdout:node-exporter.b smithi186 *:9100 running (54s) 14s ago 22m 14.6M - 1.5.0 0da6a335fe13 d7e0b3d20191 2024-03-22T05:42:44.565 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0 smithi005 running (26m) 13s ago 26m 53.5M 2993M 17.2.0 e1d6a67b021e ef5c7053dd37 2024-03-22T05:42:44.565 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1 smithi005 running (25m) 13s ago 25m 56.5M 2993M 17.2.0 e1d6a67b021e a1e7ee1017b6 2024-03-22T05:42:44.565 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2 smithi005 running (25m) 13s ago 25m 48.5M 2993M 17.2.0 e1d6a67b021e 462ea63ad0e5 2024-03-22T05:42:44.566 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3 smithi005 running (25m) 13s ago 25m 56.5M 2993M 17.2.0 e1d6a67b021e b25432a7239b 2024-03-22T05:42:44.566 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4 smithi186 running (24m) 14s ago 24m 56.1M 3248M 17.2.0 e1d6a67b021e cd963671f340 2024-03-22T05:42:44.566 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5 smithi186 running (24m) 14s ago 24m 51.6M 3248M 17.2.0 e1d6a67b021e f98c87f1a2cd 2024-03-22T05:42:44.566 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6 smithi186 running (24m) 14s ago 24m 51.7M 3248M 17.2.0 e1d6a67b021e 704ac49c68c4 2024-03-22T05:42:44.566 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7 smithi186 running (23m) 14s ago 23m 54.1M 3248M 17.2.0 e1d6a67b021e f415b90edf63 2024-03-22T05:42:44.566 INFO:teuthology.orchestra.run.smithi005.stdout:prometheus.a smithi186 *:9095 running (35s) 14s ago 22m 59.7M - 2.43.0 a07b618ecd1d a5a043588d6d 2024-03-22T05:42:44.566 INFO:teuthology.orchestra.run.smithi005.stdout:rgw.foo.smithi005.exdhqu smithi005 *:8000 running (21m) 13s ago 21m 78.8M - 17.2.0 e1d6a67b021e 6bb27fe11553 2024-03-22T05:42:44.566 INFO:teuthology.orchestra.run.smithi005.stdout:rgw.foo.smithi186.mcgodk smithi186 *:8000 running (21m) 14s ago 21m 77.1M - 17.2.0 e1d6a67b021e 19b5c20a34f7 2024-03-22T05:42:44.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:44 smithi005 ceph-mon[24793]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 616 B/s rd, 0 op/s 2024-03-22T05:42:44.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:44 smithi005 ceph-mon[24793]: from='client.25423 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:42:44.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:44 smithi005 ceph-mon[24793]: from='client.25429 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:42:44.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:44 smithi005 ceph-mon[24793]: from='client.25435 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:42:44.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:44 smithi005 ceph-mon[29523]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 616 B/s rd, 0 op/s 2024-03-22T05:42:44.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:44 smithi005 ceph-mon[29523]: from='client.25423 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:42:44.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:44 smithi005 ceph-mon[29523]: from='client.25429 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:42:44.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:44 smithi005 ceph-mon[29523]: from='client.25435 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:42:44.976 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-03-22T05:42:44.976 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": { 2024-03-22T05:42:44.976 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-03-22T05:42:44.976 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:42:44.976 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr": { 2024-03-22T05:42:44.976 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev)": 2 2024-03-22T05:42:44.976 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:42:44.976 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": { 2024-03-22T05:42:44.976 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-22T05:42:44.976 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:42:44.977 INFO:teuthology.orchestra.run.smithi005.stdout: "mds": {}, 2024-03-22T05:42:44.977 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw": { 2024-03-22T05:42:44.977 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-22T05:42:44.977 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:42:44.977 INFO:teuthology.orchestra.run.smithi005.stdout: "overall": { 2024-03-22T05:42:44.977 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-03-22T05:42:44.977 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev)": 2 2024-03-22T05:42:44.977 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-03-22T05:42:44.977 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-03-22T05:42:45.035 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:44 smithi186 ceph-mon[25720]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 616 B/s rd, 0 op/s 2024-03-22T05:42:45.035 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:44 smithi186 ceph-mon[25720]: from='client.25423 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:42:45.035 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:44 smithi186 ceph-mon[25720]: from='client.25429 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:42:45.035 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:44 smithi186 ceph-mon[25720]: from='client.25435 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:42:45.317 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-03-22T05:42:45.317 INFO:teuthology.orchestra.run.smithi005.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9", 2024-03-22T05:42:45.317 INFO:teuthology.orchestra.run.smithi005.stdout: "in_progress": true, 2024-03-22T05:42:45.317 INFO:teuthology.orchestra.run.smithi005.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-03-22T05:42:45.318 INFO:teuthology.orchestra.run.smithi005.stdout: "services_complete": [ 2024-03-22T05:42:45.318 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr" 2024-03-22T05:42:45.318 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-03-22T05:42:45.318 INFO:teuthology.orchestra.run.smithi005.stdout: "progress": "2/2 daemons upgraded", 2024-03-22T05:42:45.318 INFO:teuthology.orchestra.run.smithi005.stdout: "message": "Currently upgrading grafana daemons", 2024-03-22T05:42:45.318 INFO:teuthology.orchestra.run.smithi005.stdout: "is_paused": false 2024-03-22T05:42:45.318 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-03-22T05:42:45.981 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:45 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/2585178768' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:45.981 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:45 smithi186 ceph-mon[25720]: from='client.25447 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:42:45.981 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:45 smithi186 ceph-mon[25720]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:42:46.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:45 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/2585178768' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:46.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:45 smithi005 ceph-mon[24793]: from='client.25447 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:42:46.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:45 smithi005 ceph-mon[24793]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:42:46.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:45 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/2585178768' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:46.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:45 smithi005 ceph-mon[29523]: from='client.25447 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:42:46.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:45 smithi005 ceph-mon[29523]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:42:47.452 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:46 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:42:46] "GET /metrics HTTP/1.1" 200 37976 "" "Prometheus/2.43.0" 2024-03-22T05:42:48.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:48 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:48.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:48 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:48.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:48 smithi005 ceph-mon[24793]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:42:48.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:48 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:48.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:48 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:48.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:48 smithi005 ceph-mon[29523]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:42:48.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:48 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:48.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:48 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:48.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:48 smithi186 ceph-mon[25720]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:42:49.400 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:49 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:49.400 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:49 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:49.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:49 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:49.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:49 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:49.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:49 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:49.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:49 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:50.295 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:50 smithi186 ceph-mon[25720]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:42:50.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:50 smithi005 ceph-mon[24793]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:42:50.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:50 smithi005 ceph-mon[29523]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:42:52.540 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:52 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:52.540 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:52 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:52.540 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:52 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:42:52.540 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:52 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:42:52.540 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:52 smithi005 ceph-mon[24793]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 869 B/s rd, 0 op/s 2024-03-22T05:42:52.540 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:52 smithi005 ceph-mon[24793]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-22T05:42:52.540 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:52 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:52.540 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:52 smithi005 ceph-mon[24793]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:42:52.540 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:52 smithi005 ceph-mon[24793]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:42:52.540 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:52 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:42:52.540 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:52 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:42:52.541 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:52 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:42:52.541 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:52 smithi005 ceph-mon[24793]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:42:52.541 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:52 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:52.541 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:52 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:52.541 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:52 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:42:52.541 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:52 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:42:52.541 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:52 smithi005 ceph-mon[29523]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 869 B/s rd, 0 op/s 2024-03-22T05:42:52.541 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:52 smithi005 ceph-mon[29523]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-22T05:42:52.541 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:52 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:52.541 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:52 smithi005 ceph-mon[29523]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:42:52.541 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:52 smithi005 ceph-mon[29523]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:42:52.541 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:52 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:42:52.542 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:52 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:42:52.542 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:52 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:42:52.542 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:52 smithi005 ceph-mon[29523]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:42:52.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:52 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:52.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:52 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:52.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:52 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:42:52.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:52 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:42:52.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:52 smithi186 ceph-mon[25720]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 869 B/s rd, 0 op/s 2024-03-22T05:42:52.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:52 smithi186 ceph-mon[25720]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-22T05:42:52.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:52 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:52.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:52 smithi186 ceph-mon[25720]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:42:52.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:52 smithi186 ceph-mon[25720]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:42:52.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:52 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:42:52.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:52 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:42:52.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:52 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:42:52.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:52 smithi186 ceph-mon[25720]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:42:53.702 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:42:53.368+0000 7f487c563700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:42:53.702 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:42:53.702 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:42:53.703 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:42:53.703 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:42:53.703 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:53.703 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:53.703 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:42:53.703 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:42:53.703 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:42:53.703 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:42:53.703 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:42:53.703 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:42:53.703 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:42:53.703 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:42:53.703 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:42:53.703 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:42:53.704 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:42:53.704 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:42:53.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[24793]: Health check update: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-22T05:42:53.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:42:53.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-22T05:42:53.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:42:53.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:53.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:53.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:53.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:53.705 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:53.705 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:53.705 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:53.705 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:53.705 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:53.705 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:53.705 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:53.705 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:53.705 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:53.705 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:53.705 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:53.705 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-22T05:42:53.705 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-22T05:42:53.706 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-22T05:42:53.706 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:42:53.706 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:42:53.706 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-22T05:42:53.706 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-22T05:42:53.706 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-22T05:42:53.706 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[29523]: Health check update: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-22T05:42:53.706 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:42:53.707 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-22T05:42:53.707 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:42:53.707 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:53.707 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:53.707 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:53.707 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:53.707 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:53.707 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:53.707 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:53.707 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:53.707 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:53.707 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:53.707 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:53.708 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:53.708 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:53.708 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:53.708 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:53.708 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-22T05:42:53.708 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-22T05:42:53.708 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-22T05:42:53.708 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:42:53.708 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:42:53.708 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-22T05:42:53.708 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-22T05:42:53.708 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-22T05:42:53.709 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:53.709 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:53.709 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:42:53.709 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:42:53.709 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:42:53.709 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:42:53.709 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:42:53.709 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:42:53.709 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:42:53.709 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:42:53.709 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return future.result(timeout) 2024-03-22T05:42:53.710 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:42:53.710 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.__get_result() 2024-03-22T05:42:53.710 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:42:53.710 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: raise self._exception 2024-03-22T05:42:53.710 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:42:53.710 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: config_blobs=daemon_spec.final_config, 2024-03-22T05:42:53.710 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:42:53.710 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:42:53.710 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:42:53.710 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:42:53.710 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:42:53.710 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:42:53.710 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:42:53.711 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:53.711 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-22T05:42:53.711 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-22T05:42:53.711 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-22T05:42:53.711 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-22T05:42:53.711 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-22T05:42:53.711 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-22T05:42:53.712 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-22T05:42:53.712 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-22T05:42:53.712 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-22T05:42:53.712 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-22T05:42:53.712 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-22T05:42:53.712 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:42:53.712 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:42:53.712 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-22T05:42:53.712 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-22T05:42:53.712 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-22T05:42:53.712 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-22T05:42:53.712 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-22T05:42:53.712 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-22T05:42:53.712 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-22T05:42:53.713 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-22T05:42:53.713 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-22T05:42:53.713 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-22T05:42:53.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-22T05:42:53.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-22T05:42:53.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-22T05:42:53.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-22T05:42:53.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-22T05:42:53.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-22T05:42:53.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-22T05:42:53.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-22T05:42:53.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-22T05:42:53.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-22T05:42:53.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-22T05:42:53.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:42:53.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:42:53.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-22T05:42:53.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-22T05:42:53.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-22T05:42:53.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-22T05:42:53.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-22T05:42:53.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-22T05:42:53.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-22T05:42:53.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-22T05:42:53.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-22T05:42:53.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-22T05:42:53.715 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:53.715 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:42:53.715 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:42:53.715 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:42:53.716 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:42:53.716 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:42:53.716 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:42:53.716 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:42:53.716 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:42:53.716 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:42:53.716 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:42:53.716 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:42:53.716 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:42:53.716 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:53.716 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:53 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:53.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:53 smithi186 ceph-mon[25720]: Health check update: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-22T05:42:53.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:53 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:42:53.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:53 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-22T05:42:53.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:53 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:42:53.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:53 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:53.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:53 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:53.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:53 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:53.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:53 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:53.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:53 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:53.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:53 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:53.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:53 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:53.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:53 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:53.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:53 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:53.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:53 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:53.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:53 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:53.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:53 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:53.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:53 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:53.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:53 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:42:53.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:53 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:53.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:53 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-22T05:42:53.796 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:53 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-22T05:42:53.796 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:53 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-22T05:42:53.796 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:53 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:42:53.796 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:53 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:42:53.796 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:53 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-22T05:42:53.796 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:53 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-22T05:42:53.796 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:53 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-22T05:42:53.796 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:53 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-22T05:42:53.796 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:53 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-22T05:42:53.796 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:53 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-22T05:42:53.796 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:53 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-22T05:42:53.796 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:53 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-22T05:42:53.796 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:53 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-22T05:42:53.797 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:53 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-22T05:42:53.797 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:53 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-22T05:42:53.797 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:53 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-22T05:42:53.797 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:53 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-22T05:42:53.797 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:53 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-22T05:42:53.797 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:53 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:42:53.797 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:53 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:42:53.797 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:53 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-22T05:42:53.797 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:53 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-22T05:42:53.797 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:53 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-22T05:42:53.797 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:53 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-22T05:42:53.797 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:53 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-22T05:42:53.797 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:53 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-22T05:42:53.798 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:53 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-22T05:42:53.798 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:53 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-22T05:42:53.798 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:53 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-22T05:42:53.798 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:53 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-22T05:42:54.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:54 smithi005 ceph-mon[24793]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:42:54.702 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:42:54.702 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:42:54.702 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:42:54.702 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:42:54.702 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:54.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:54.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:42:54.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:42:54.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:42:54.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:42:54.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:42:54.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:42:54.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:42:54.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:42:54.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:42:54.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:42:54.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:42:54.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:42:54.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:54.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:54.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:42:54.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:42:54.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:42:54.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:42:54.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:42:54.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:42:54.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:42:54.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:42:54.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: return future.result(timeout) 2024-03-22T05:42:54.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:42:54.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.__get_result() 2024-03-22T05:42:54.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:42:54.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: raise self._exception 2024-03-22T05:42:54.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:42:54.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:42:54.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:42:54.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:42:54.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:42:54.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:42:54.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:42:54.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:42:54.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:42:54.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:54.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:54.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:42:54.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:42:54.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:42:54.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:42:54.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:42:54.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:42:54.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:42:54.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:42:54.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:42:54.706 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:54 smithi005 ceph-mon[29523]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:42:54.706 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:42:54.706 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:42:54.706 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:42:54.706 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:42:54.706 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:54.706 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:54.706 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:42:54.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:42:54.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:42:54.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:42:54.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:42:54.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:42:54.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:42:54.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:42:54.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:42:54.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:42:54.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:42:54.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:42:54.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:54.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:54.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:42:54.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:42:54.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:42:54.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:42:54.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:42:54.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:42:54.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:42:54.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:42:54.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: return future.result(timeout) 2024-03-22T05:42:54.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:42:54.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.__get_result() 2024-03-22T05:42:54.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:42:54.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: raise self._exception 2024-03-22T05:42:54.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:42:54.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:42:54.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:42:54.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:42:54.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:42:54.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:42:54.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:42:54.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:42:54.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:42:54.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:54.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:54.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:42:54.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:42:54.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:42:54.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:42:54.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:42:54.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:42:54.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:42:54.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:42:54.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:42:54.711 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:42:54.711 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:42:54.711 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:42:54.711 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:54.711 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:54.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:54 smithi005 ceph-mon[24793]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-22T05:42:54.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:54 smithi005 ceph-mon[24793]: Upgrade: Finalizing container_image settings 2024-03-22T05:42:54.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:54 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-22T05:42:54.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:54 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:42:54.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:54 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:42:54.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:54 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:42:54.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:54 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:42:54.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:54 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:42:54.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:54 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:42:54.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:54 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:42:54.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:54 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:42:54.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:54 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:42:54.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:54 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:42:54.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:54 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:42:54.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:54 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:42:54.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:54 smithi005 ceph-mon[24793]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-22T05:42:54.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:54 smithi005 ceph-mon[24793]: Upgrade: Complete! 2024-03-22T05:42:54.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:54 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-22T05:42:54.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:54 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-22T05:42:54.712 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:42:54.713 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:42:54.713 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:42:54.713 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:54.713 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:54.713 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:54 smithi005 ceph-mon[29523]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-22T05:42:54.713 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:54 smithi005 ceph-mon[29523]: Upgrade: Finalizing container_image settings 2024-03-22T05:42:54.713 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:54 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-22T05:42:54.713 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:54 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:42:54.713 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:54 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:42:54.713 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:54 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:42:54.713 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:54 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:42:54.713 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:54 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:42:54.713 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:54 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:42:54.713 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:54 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:42:54.714 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:54 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:42:54.714 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:54 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:42:54.714 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:54 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:42:54.714 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:54 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:42:54.714 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:54 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:42:54.714 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:54 smithi005 ceph-mon[29523]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-22T05:42:54.714 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:54 smithi005 ceph-mon[29523]: Upgrade: Complete! 2024-03-22T05:42:54.714 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:54 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-22T05:42:54.714 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:54 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-22T05:42:54.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:54 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-22T05:42:54.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:54 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:42:54.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:54 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:42:54.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:54 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:42:54.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:54 smithi005 ceph-mon[24793]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 657 B/s rd, 0 op/s 2024-03-22T05:42:54.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:54 smithi005 ceph-mon[24793]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 885 B/s rd, 0 op/s 2024-03-22T05:42:54.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:54 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:54.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:54 smithi005 ceph-mon[24793]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:42:54.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:54 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:42:54.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:54 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:42:54.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:54 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:42:54.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:54 smithi005 ceph-mon[24793]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:42:54.715 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:54 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-22T05:42:54.715 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:54 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:42:54.716 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:54 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:42:54.716 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:54 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:42:54.716 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:54 smithi005 ceph-mon[29523]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 657 B/s rd, 0 op/s 2024-03-22T05:42:54.716 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:54 smithi005 ceph-mon[29523]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 885 B/s rd, 0 op/s 2024-03-22T05:42:54.716 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:54 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:54.716 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:54 smithi005 ceph-mon[29523]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:42:54.716 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:54 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:42:54.716 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:54 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:42:54.716 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:54 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:42:54.716 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:54 smithi005 ceph-mon[29523]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:42:54.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:54 smithi186 ceph-mon[25720]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:42:54.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:42:54.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:42:54.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:42:54.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:42:54.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:54.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:54.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:42:54.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:42:54.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:42:54.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:42:54.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:42:54.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:42:54.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:42:54.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:42:54.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:42:54.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:42:54.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:42:54.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:42:54.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:54.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:54.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:42:54.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:42:54.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:42:54.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:42:54.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:42:54.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:42:54.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:42:54.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:42:54.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: return future.result(timeout) 2024-03-22T05:42:54.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:42:54.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.__get_result() 2024-03-22T05:42:54.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:42:54.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: raise self._exception 2024-03-22T05:42:54.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:42:54.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:42:54.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:42:54.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:42:54.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:42:54.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:42:54.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:42:54.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:42:54.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:42:54.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:54.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:54.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:42:54.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:42:54.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:42:54.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:42:54.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:42:54.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:42:54.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:42:54.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:42:54.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:42:54.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:42:54.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:42:54.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:42:54.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:54.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:54.798 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:54 smithi186 ceph-mon[25720]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-22T05:42:54.798 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:54 smithi186 ceph-mon[25720]: Upgrade: Finalizing container_image settings 2024-03-22T05:42:54.798 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:54 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-22T05:42:54.799 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:54 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:42:54.799 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:54 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:42:54.799 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:54 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:42:54.799 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:54 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:42:54.799 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:54 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:42:54.799 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:54 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:42:54.799 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:54 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:42:54.799 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:54 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:42:54.799 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:54 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:42:54.799 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:54 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:42:54.799 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:54 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:42:54.799 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:54 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:42:54.799 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:54 smithi186 ceph-mon[25720]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-22T05:42:54.799 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:54 smithi186 ceph-mon[25720]: Upgrade: Complete! 2024-03-22T05:42:54.799 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:54 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-22T05:42:54.800 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:54 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-22T05:42:54.800 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:54 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-22T05:42:54.800 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:54 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:42:54.800 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:54 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:42:54.800 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:54 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:42:54.800 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:54 smithi186 ceph-mon[25720]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 657 B/s rd, 0 op/s 2024-03-22T05:42:54.800 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:54 smithi186 ceph-mon[25720]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 885 B/s rd, 0 op/s 2024-03-22T05:42:54.800 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:54 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:54.800 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:54 smithi186 ceph-mon[25720]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:42:54.800 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:54 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:42:54.800 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:54 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:42:54.800 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:54 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:42:54.800 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:54 smithi186 ceph-mon[25720]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:42:55.452 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:55 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:42:55.239+0000 7f487c563700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:42:55.453 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:55 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:42:55.453 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:55 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:42:55.453 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:55 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:42:55.453 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:55 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:42:55.453 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:55 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:55.453 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:55 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:55.453 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:55 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:42:55.453 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:55 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:42:55.453 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:55 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:42:55.453 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:55 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:42:55.453 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:55 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:42:55.453 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:55 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:42:55.454 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:55 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:42:55.454 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:55 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:42:55.454 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:55 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:42:55.454 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:55 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:42:55.454 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:55 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:42:55.454 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:55 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:42:55.454 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:55 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:55.454 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:55 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:55.455 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:55 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:42:55.455 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:55 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:42:55.455 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:55 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:42:55.455 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:55 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:42:55.455 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:55 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:42:55.455 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:55 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:42:55.455 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:55 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:42:55.455 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:55 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:42:55.455 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:55 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return future.result(timeout) 2024-03-22T05:42:55.455 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:55 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:42:55.455 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:55 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.__get_result() 2024-03-22T05:42:55.455 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:55 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:42:55.456 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:55 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: raise self._exception 2024-03-22T05:42:55.456 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:55 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:42:55.456 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:55 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: config_blobs=daemon_spec.final_config, 2024-03-22T05:42:55.456 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:55 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:42:55.456 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:55 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:42:55.456 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:55 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:42:55.456 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:55 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:42:55.456 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:55 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:42:55.456 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:55 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:42:55.456 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:55 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:42:55.456 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:55 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:55.456 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:55 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:55.456 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:55 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:42:55.456 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:55 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:42:55.457 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:55 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:42:55.457 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:55 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:42:55.457 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:55 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:42:55.457 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:55 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:42:55.457 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:55 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:42:55.457 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:55 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:42:55.457 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:55 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:42:55.457 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:55 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:42:55.457 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:55 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:42:55.457 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:55 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:42:55.457 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:55 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:55.457 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:55 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:55.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:55 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:42:55.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:55 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:42:55.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:55 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:42:55.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:55 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:55.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:55 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:42:55.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:55 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:42:55.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:55 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:42:55.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:55 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:42:55.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:55 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:42:55.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:55 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:42:55.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:55 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:55.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:55 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:42:55.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:55 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:42:55.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:55 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:42:55.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:55 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:42:55.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:55 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:42:55.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:55 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:42:55.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:55 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:55.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:55 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:42:55.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:55 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:42:55.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:55 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:42:56.694 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:56 smithi005 ceph-mon[24793]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:42:56.694 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:42:56.694 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:42:56.694 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:42:56.694 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:42:56.694 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:56.694 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:56.694 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:42:56.694 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:42:56.694 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:42:56.694 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:42:56.695 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:42:56.695 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:42:56.695 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:42:56.695 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:42:56.695 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:42:56.695 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:42:56.695 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:42:56.695 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:42:56.695 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:56.695 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:56.695 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:42:56.695 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:42:56.695 INFO:journalctl@ceph.mon.a.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:42:56.695 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:42:56.696 INFO:journalctl@ceph.mon.a.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:42:56.696 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:42:56.696 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:42:56.696 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:42:56.696 INFO:journalctl@ceph.mon.a.smithi005.stdout: return future.result(timeout) 2024-03-22T05:42:56.696 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:42:56.696 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.__get_result() 2024-03-22T05:42:56.696 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:42:56.696 INFO:journalctl@ceph.mon.a.smithi005.stdout: raise self._exception 2024-03-22T05:42:56.696 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:42:56.696 INFO:journalctl@ceph.mon.a.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:42:56.696 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:42:56.696 INFO:journalctl@ceph.mon.a.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:42:56.696 INFO:journalctl@ceph.mon.a.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:42:56.696 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:42:56.696 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:42:56.697 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:42:56.697 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:42:56.697 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:56.697 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:56.697 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:42:56.697 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:42:56.697 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:42:56.697 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:42:56.697 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:42:56.697 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:42:56.697 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:42:56.697 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:42:56.697 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:42:56.698 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:56 smithi005 ceph-mon[29523]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:42:56.698 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:42:56.698 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:42:56.698 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:42:56.698 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:42:56.698 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:56.698 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:56.698 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:42:56.698 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:42:56.698 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:42:56.698 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:42:56.699 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:42:56.699 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:42:56.699 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:42:56.699 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:42:56.699 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:42:56.699 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:42:56.699 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:42:56.699 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:42:56.699 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:56.699 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:56.699 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:42:56.700 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:42:56.700 INFO:journalctl@ceph.mon.c.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:42:56.700 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:42:56.700 INFO:journalctl@ceph.mon.c.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:42:56.700 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:42:56.700 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:42:56.700 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:42:56.700 INFO:journalctl@ceph.mon.c.smithi005.stdout: return future.result(timeout) 2024-03-22T05:42:56.700 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:42:56.700 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.__get_result() 2024-03-22T05:42:56.700 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:42:56.700 INFO:journalctl@ceph.mon.c.smithi005.stdout: raise self._exception 2024-03-22T05:42:56.700 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:42:56.700 INFO:journalctl@ceph.mon.c.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:42:56.700 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:42:56.701 INFO:journalctl@ceph.mon.c.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:42:56.701 INFO:journalctl@ceph.mon.c.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:42:56.701 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:42:56.701 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:42:56.701 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:42:56.701 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:42:56.701 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:56.701 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:56.701 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:42:56.701 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:42:56.701 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:42:56.701 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:42:56.701 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:42:56.701 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:42:56.701 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:42:56.702 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:42:56.702 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:42:56.702 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:42:56.702 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:42:56.702 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:42:56.702 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:56.702 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:56.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:56 smithi005 ceph-mon[24793]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 919 B/s rd, 0 op/s 2024-03-22T05:42:56.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:56 smithi005 ceph-mon[24793]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-22T05:42:56.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:56 smithi005 ceph-mon[24793]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:42:56.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:56 smithi005 ceph-mon[24793]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:42:56.703 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:42:56.703 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:42:56.703 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:42:56.703 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:56.703 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:56.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:56 smithi005 ceph-mon[29523]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 919 B/s rd, 0 op/s 2024-03-22T05:42:56.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:56 smithi005 ceph-mon[29523]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-22T05:42:56.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:56 smithi005 ceph-mon[29523]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:42:56.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:56 smithi005 ceph-mon[29523]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:42:56.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:56 smithi186 ceph-mon[25720]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:42:56.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:42:56.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:42:56.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:42:56.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:42:56.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:56.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:56.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:42:56.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:42:56.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:42:56.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:42:56.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:42:56.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:42:56.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:42:56.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:42:56.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:42:56.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:42:56.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:42:56.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:42:56.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:56.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:56.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:42:56.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:42:56.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:42:56.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:42:56.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:42:56.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:42:56.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:42:56.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:42:56.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: return future.result(timeout) 2024-03-22T05:42:56.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:42:56.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.__get_result() 2024-03-22T05:42:56.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:42:56.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: raise self._exception 2024-03-22T05:42:56.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:42:56.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:42:56.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:42:56.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:42:56.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:42:56.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:42:56.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:42:56.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:42:56.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:42:56.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:56.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:56.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:42:56.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:42:56.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:42:56.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:42:56.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:42:56.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:42:56.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:42:56.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:42:56.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:42:56.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:42:56.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:42:56.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:42:56.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:56.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:56.798 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:56 smithi186 ceph-mon[25720]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 919 B/s rd, 0 op/s 2024-03-22T05:42:56.798 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:56 smithi186 ceph-mon[25720]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-22T05:42:56.798 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:56 smithi186 ceph-mon[25720]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:42:56.798 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:56 smithi186 ceph-mon[25720]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:42:57.452 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:56 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:42:56] "GET /metrics HTTP/1.1" 200 37974 "" "Prometheus/2.43.0" 2024-03-22T05:42:57.452 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:42:57.105+0000 7f487c563700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:42:57.452 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:42:57.452 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:42:57.453 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:42:57.453 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:42:57.453 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:57.453 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:57.453 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:42:57.453 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:42:57.453 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:42:57.453 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:42:57.453 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:42:57.453 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:42:57.453 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:42:57.453 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:42:57.453 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:42:57.453 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:42:57.453 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:42:57.454 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:42:57.454 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:57.454 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:57.454 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:42:57.454 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:42:57.455 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:42:57.455 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:42:57.455 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:42:57.455 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:42:57.455 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:42:57.455 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:42:57.455 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return future.result(timeout) 2024-03-22T05:42:57.455 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:42:57.455 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.__get_result() 2024-03-22T05:42:57.455 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:42:57.455 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: raise self._exception 2024-03-22T05:42:57.455 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:42:57.455 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: config_blobs=daemon_spec.final_config, 2024-03-22T05:42:57.455 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:42:57.455 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:42:57.456 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:42:57.456 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:42:57.456 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:42:57.456 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:42:57.456 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:42:57.456 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:57.456 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:57.456 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:42:57.456 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:42:57.456 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:42:57.456 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:42:57.456 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:42:57.456 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:42:57.456 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:42:57.457 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:42:57.457 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:42:57.457 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:42:57.457 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:42:57.457 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:42:57.457 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:57.457 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:42:57 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:58.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:58 smithi186 ceph-mon[25720]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:42:58.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:42:58.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:42:58.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:42:58.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:42:58.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:58.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:58.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:42:58.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:42:58.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:42:58.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:42:58.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:42:58.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:42:58.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:42:58.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:42:58.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:42:58.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:42:58.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:42:58.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:42:58.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:58.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:58.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:42:58.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:42:58.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:42:58.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:42:58.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:42:58.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:42:58.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:42:58.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:42:58.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: return future.result(timeout) 2024-03-22T05:42:58.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:42:58.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.__get_result() 2024-03-22T05:42:58.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:42:58.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: raise self._exception 2024-03-22T05:42:58.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:42:58.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:42:58.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:42:58.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:42:58.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:42:58.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:42:58.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:42:58.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:42:58.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:42:58.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:58.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:58.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:42:58.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:42:58.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:42:58.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:42:58.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:42:58.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:42:58.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:42:58.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:42:58.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:42:58.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:42:58.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:42:58.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:42:58.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:58.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:58.798 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:58 smithi186 ceph-mon[25720]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 884 B/s rd, 0 op/s 2024-03-22T05:42:58.798 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:42:58 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:58.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:58 smithi005 ceph-mon[29523]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:42:58.952 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:42:58.952 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:42:58.952 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:42:58.952 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:42:58.953 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:58.953 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:58.953 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:42:58.953 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:42:58.953 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:42:58.953 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:42:58.953 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:42:58.953 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:42:58.953 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:42:58.953 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:42:58.953 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:42:58.953 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:42:58.953 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:42:58.954 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:42:58.954 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:58.954 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:58.954 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:42:58.954 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:42:58.954 INFO:journalctl@ceph.mon.c.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:42:58.954 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:42:58.954 INFO:journalctl@ceph.mon.c.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:42:58.954 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:42:58.954 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:42:58.954 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:42:58.954 INFO:journalctl@ceph.mon.c.smithi005.stdout: return future.result(timeout) 2024-03-22T05:42:58.954 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:42:58.954 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.__get_result() 2024-03-22T05:42:58.955 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:42:58.955 INFO:journalctl@ceph.mon.c.smithi005.stdout: raise self._exception 2024-03-22T05:42:58.955 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:42:58.955 INFO:journalctl@ceph.mon.c.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:42:58.955 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:42:58.955 INFO:journalctl@ceph.mon.c.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:42:58.955 INFO:journalctl@ceph.mon.c.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:42:58.955 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:42:58.955 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:42:58.955 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:42:58.955 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:42:58.955 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:58.955 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:58.955 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:42:58.956 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:42:58.956 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:42:58.956 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:42:58.956 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:42:58.956 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:42:58.956 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:42:58.956 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:42:58.956 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:42:58.957 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:58 smithi005 ceph-mon[24793]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:42:58.957 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:42:58.957 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:42:58.957 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:42:58.957 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:42:58.957 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:58.957 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:58.958 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:42:58.958 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:42:58.958 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:42:58.958 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:42:58.958 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:42:58.958 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:42:58.958 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:42:58.958 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:42:58.958 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:42:58.958 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:42:58.958 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:42:58.959 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:42:58.959 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:58.959 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:58.959 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:42:58.959 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:42:58.959 INFO:journalctl@ceph.mon.a.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:42:58.959 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:42:58.959 INFO:journalctl@ceph.mon.a.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:42:58.959 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:42:58.959 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:42:58.959 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:42:58.959 INFO:journalctl@ceph.mon.a.smithi005.stdout: return future.result(timeout) 2024-03-22T05:42:58.960 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:42:58.960 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.__get_result() 2024-03-22T05:42:58.960 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:42:58.960 INFO:journalctl@ceph.mon.a.smithi005.stdout: raise self._exception 2024-03-22T05:42:58.960 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:42:58.960 INFO:journalctl@ceph.mon.a.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:42:58.960 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:42:58.960 INFO:journalctl@ceph.mon.a.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:42:58.960 INFO:journalctl@ceph.mon.a.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:42:58.960 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:42:58.960 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:42:58.960 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:42:58.960 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:42:58.960 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:58.961 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:58.961 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:42:58.961 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:42:58.961 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:42:58.961 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:42:58.961 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:42:58.961 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:42:58.961 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:42:58.961 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:42:58.961 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:42:58.961 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:42:58.962 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:42:58.962 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:42:58.962 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:58.962 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:58.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:58 smithi005 ceph-mon[24793]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 884 B/s rd, 0 op/s 2024-03-22T05:42:58.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:42:58 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:42:58.962 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:42:58.962 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:42:58.962 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:42:58.962 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:42:58.962 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:42:58.963 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:58 smithi005 ceph-mon[29523]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 884 B/s rd, 0 op/s 2024-03-22T05:42:58.963 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:42:58 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:43:00.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:00 smithi186 ceph-mon[25720]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-22T05:43:00.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:00 smithi005 ceph-mon[29523]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-22T05:43:00.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:00 smithi005 ceph-mon[24793]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-22T05:43:02.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:02 smithi186 ceph-mon[25720]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-22T05:43:02.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:02 smithi186 ceph-mon[25720]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:43:02.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:02 smithi005 ceph-mon[24793]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-22T05:43:02.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:02 smithi005 ceph-mon[24793]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:43:02.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:02 smithi005 ceph-mon[29523]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-22T05:43:02.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:02 smithi005 ceph-mon[29523]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:43:04.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:04 smithi186 ceph-mon[25720]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-22T05:43:04.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:04 smithi005 ceph-mon[24793]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-22T05:43:04.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:04 smithi005 ceph-mon[29523]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-22T05:43:06.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:06 smithi186 ceph-mon[25720]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:43:06.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:06 smithi005 ceph-mon[24793]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:43:06.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:06 smithi005 ceph-mon[29523]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:43:07.213 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:43:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:43:06] "GET /metrics HTTP/1.1" 200 37974 "" "Prometheus/2.43.0" 2024-03-22T05:43:08.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:08 smithi186 ceph-mon[25720]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:43:08.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:08 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:43:08.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:08 smithi005 ceph-mon[24793]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:43:08.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:08 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:43:08.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:08 smithi005 ceph-mon[29523]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:43:08.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:08 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:43:10.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:10 smithi186 ceph-mon[25720]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:43:10.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:10 smithi005 ceph-mon[29523]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:43:10.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:10 smithi005 ceph-mon[24793]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:43:12.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:12 smithi186 ceph-mon[25720]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:43:12.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:12 smithi186 ceph-mon[25720]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:43:12.873 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:12 smithi005 ceph-mon[24793]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:43:12.873 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:12 smithi005 ceph-mon[24793]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:43:12.874 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:12 smithi005 ceph-mon[29523]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:43:12.874 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:12 smithi005 ceph-mon[29523]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:43:14.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:14 smithi186 ceph-mon[25720]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:43:14.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:14 smithi005 ceph-mon[29523]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:43:14.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:14 smithi005 ceph-mon[24793]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:43:16.262 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph versions | jq -e '"'"'.mgr | length == 1'"'"'' 2024-03-22T05:43:16.578 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:16 smithi005 ceph-mon[24793]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:43:16.578 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:16 smithi005 ceph-mon[24793]: from='client.25453 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:43:16.578 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:16 smithi005 ceph-mon[29523]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:43:16.578 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:16 smithi005 ceph-mon[29523]: from='client.25453 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:43:16.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:16 smithi186 ceph-mon[25720]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:43:16.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:16 smithi186 ceph-mon[25720]: from='client.25453 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:43:17.202 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:43:16 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:43:16] "GET /metrics HTTP/1.1" 200 37976 "" "Prometheus/2.43.0" 2024-03-22T05:43:18.526 INFO:teuthology.orchestra.run.smithi005.stdout:true 2024-03-22T05:43:18.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:18 smithi186 ceph-mon[25720]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:43:18.923 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:18 smithi005 ceph-mon[24793]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:43:18.923 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:18 smithi005 ceph-mon[29523]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:43:19.170 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph versions | jq -e '"'"'.mgr | keys'"'"' | grep $sha1' 2024-03-22T05:43:19.790 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:19 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/1392138082' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:43:19.790 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:19 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/1392138082' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:43:19.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:19 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/1392138082' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:43:20.559 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:20 smithi005 ceph-mon[29523]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:43:20.559 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:20 smithi005 ceph-mon[24793]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:43:20.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:20 smithi186 ceph-mon[25720]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:43:21.346 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev)" 2024-03-22T05:43:21.617 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:21 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/2928055969' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:43:21.618 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:21 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/2928055969' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:43:21.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:21 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/2928055969' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:43:22.152 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 2'"'"'' 2024-03-22T05:43:22.733 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:22 smithi005 ceph-mon[24793]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:43:22.733 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:22 smithi005 ceph-mon[24793]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:43:22.733 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:22 smithi005 ceph-mon[29523]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:43:22.733 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:22 smithi005 ceph-mon[29523]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:43:22.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:22 smithi186 ceph-mon[25720]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:43:22.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:22 smithi186 ceph-mon[25720]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:43:23.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:23 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:43:23.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:23 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:43:23.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:23 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:43:24.250 INFO:teuthology.orchestra.run.smithi005.stdout:true 2024-03-22T05:43:24.773 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:24 smithi005 ceph-mon[24793]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:43:24.773 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:24 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/1785241035' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:43:24.773 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:24 smithi005 ceph-mon[29523]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:43:24.773 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:24 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/1785241035' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:43:24.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:24 smithi186 ceph-mon[25720]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:43:24.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:24 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/1785241035' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:43:24.797 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '"'"'.up_to_date | length == 2'"'"'' 2024-03-22T05:43:26.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:26 smithi005 ceph-mon[24793]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:43:26.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:26 smithi005 ceph-mon[29523]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:43:26.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:26 smithi186 ceph-mon[25720]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:43:27.452 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:43:26 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:43:26] "GET /metrics HTTP/1.1" 200 37987 "" "Prometheus/2.43.0" 2024-03-22T05:43:28.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:28 smithi186 ceph-mon[25720]: from='client.25477 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:43:28.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:28 smithi186 ceph-mon[25720]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:43:28.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:28 smithi005 ceph-mon[24793]: from='client.25477 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:43:28.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:28 smithi005 ceph-mon[24793]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:43:28.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:28 smithi005 ceph-mon[29523]: from='client.25477 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:43:28.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:28 smithi005 ceph-mon[29523]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:43:29.141 INFO:teuthology.orchestra.run.smithi005.stdout:true 2024-03-22T05:43:29.818 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph orch upgrade status' 2024-03-22T05:43:30.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:30 smithi005 ceph-mon[24793]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:43:30.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:30 smithi005 ceph-mon[29523]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:43:30.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:30 smithi186 ceph-mon[25720]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:43:31.825 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-03-22T05:43:31.825 INFO:teuthology.orchestra.run.smithi005.stdout: "target_image": null, 2024-03-22T05:43:31.826 INFO:teuthology.orchestra.run.smithi005.stdout: "in_progress": false, 2024-03-22T05:43:31.826 INFO:teuthology.orchestra.run.smithi005.stdout: "which": "", 2024-03-22T05:43:31.826 INFO:teuthology.orchestra.run.smithi005.stdout: "services_complete": [], 2024-03-22T05:43:31.826 INFO:teuthology.orchestra.run.smithi005.stdout: "progress": null, 2024-03-22T05:43:31.826 INFO:teuthology.orchestra.run.smithi005.stdout: "message": "", 2024-03-22T05:43:31.826 INFO:teuthology.orchestra.run.smithi005.stdout: "is_paused": false 2024-03-22T05:43:31.826 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-03-22T05:43:32.403 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph health detail' 2024-03-22T05:43:32.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:32 smithi005 ceph-mon[29523]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:43:32.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:32 smithi005 ceph-mon[29523]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:43:32.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:32 smithi005 ceph-mon[24793]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:43:32.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:32 smithi005 ceph-mon[24793]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:43:32.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:32 smithi186 ceph-mon[25720]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:43:32.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:32 smithi186 ceph-mon[25720]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:43:33.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:33 smithi186 ceph-mon[25720]: from='client.25483 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:43:33.821 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:33 smithi005 ceph-mon[24793]: from='client.25483 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:43:33.821 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:33 smithi005 ceph-mon[29523]: from='client.25483 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:43:34.655 INFO:teuthology.orchestra.run.smithi005.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-22T05:43:34.655 INFO:teuthology.orchestra.run.smithi005.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-22T05:43:34.655 INFO:teuthology.orchestra.run.smithi005.stdout: daemon iscsi.foo.smithi005.mcdwbp on smithi005 is in unknown state 2024-03-22T05:43:34.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:34 smithi186 ceph-mon[25720]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:43:34.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:34 smithi005 ceph-mon[24793]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:43:34.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:34 smithi005 ceph-mon[29523]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:43:35.288 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mon --hosts $(ceph orch ps | grep mgr.x | awk '"'"'{print $2}'"'"')' 2024-03-22T05:43:35.599 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:35 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/3856049326' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T05:43:35.600 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:35 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/3856049326' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T05:43:35.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:35 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/3856049326' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T05:43:36.560 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:36 smithi005 ceph-mon[29523]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:43:36.561 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:36 smithi005 ceph-mon[24793]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:43:36.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:36 smithi186 ceph-mon[25720]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:43:37.202 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:43:36 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:43:36] "GET /metrics HTTP/1.1" 200 37987 "" "Prometheus/2.43.0" 2024-03-22T05:43:38.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:38 smithi186 ceph-mon[25720]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:43:38.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:38 smithi186 ceph-mon[25720]: from='client.25495 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:43:38.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:38 smithi186 ceph-mon[25720]: from='client.15600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37", "daemon_types": "mon", "hosts": "smithi186", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:43:38.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:38 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:43:38.829 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:38 smithi005 ceph-mon[24793]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:43:38.830 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:38 smithi005 ceph-mon[24793]: from='client.25495 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:43:38.830 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:38 smithi005 ceph-mon[24793]: from='client.15600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37", "daemon_types": "mon", "hosts": "smithi186", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:43:38.830 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:38 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:43:38.830 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:38 smithi005 ceph-mon[29523]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:43:38.830 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:38 smithi005 ceph-mon[29523]: from='client.25495 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:43:38.830 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:38 smithi005 ceph-mon[29523]: from='client.15600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37", "daemon_types": "mon", "hosts": "smithi186", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:43:38.830 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:38 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:43:39.667 INFO:teuthology.orchestra.run.smithi005.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37 2024-03-22T05:43:40.293 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-03-22T05:43:40.890 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:40 smithi005 ceph-mon[29523]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:43:40.891 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:40 smithi005 ceph-mon[29523]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37 2024-03-22T05:43:40.891 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:40 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:43:40.891 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:40 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:43:40.891 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:40 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:43:40.891 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:40 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:43:40.891 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:40 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:43:40.891 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:40 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:43:40.891 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:40 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:43:40.891 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:40 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:43:40.891 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:40 smithi005 ceph-mon[24793]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:43:40.892 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:40 smithi005 ceph-mon[24793]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37 2024-03-22T05:43:40.892 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:40 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:43:40.892 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:40 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:43:40.892 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:40 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:43:40.892 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:40 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:43:40.892 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:40 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:43:40.892 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:40 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:43:40.892 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:40 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:43:40.892 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:40 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:43:41.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:40 smithi186 ceph-mon[25720]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:43:41.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:40 smithi186 ceph-mon[25720]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37 2024-03-22T05:43:41.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:40 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:43:41.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:40 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:43:41.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:40 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:43:41.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:40 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:43:41.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:40 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:43:41.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:40 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:43:41.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:40 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:43:41.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:40 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:43:41.945 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:41 smithi005 ceph-mon[29523]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 911 B/s rd, 0 op/s 2024-03-22T05:43:41.945 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:41 smithi005 ceph-mon[29523]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:43:41.945 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:41 smithi005 ceph-mon[29523]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:43:41.945 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:41 smithi005 ceph-mon[29523]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:43:41.945 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:41 smithi005 ceph-mon[24793]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 911 B/s rd, 0 op/s 2024-03-22T05:43:41.945 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:41 smithi005 ceph-mon[24793]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:43:41.945 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:41 smithi005 ceph-mon[24793]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:43:41.945 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:41 smithi005 ceph-mon[24793]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:43:42.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:41 smithi186 ceph-mon[25720]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 911 B/s rd, 0 op/s 2024-03-22T05:43:42.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:41 smithi186 ceph-mon[25720]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:43:42.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:41 smithi186 ceph-mon[25720]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:43:42.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:41 smithi186 ceph-mon[25720]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:43:42.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:42 smithi005 ceph-mon[24793]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:43:42.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:42 smithi005 ceph-mon[29523]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:43:43.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:42 smithi186 ceph-mon[25720]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:43:43.342 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:43:43 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:43:43.197+0000 7f487c563700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:43:43.342 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:43:43 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:43:43.342 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:43:43 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:43:43.342 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:43:43 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:43:43.343 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:43:43 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:43:43.343 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:43:43 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:43:43.343 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:43:43 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:43:43.343 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:43:43 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:43:43.343 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:43:43 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:43:43.344 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:43:43 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:43:43.344 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:43:43 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:43:43.344 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:43:43 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:43:43.344 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:43:43 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:43:43.344 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:43:43 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:43:43.345 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:43:43 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:43:43.345 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:43:43 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:43:43.345 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:43:43 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:43:43.345 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:43:43 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:43:43.345 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:43:43 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:43:43.346 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:43:43 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:43:43.346 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:43:43 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:43:43.346 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:43:43 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:43:43.346 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:43:43 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:43:43.346 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:43:43 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:43:43.346 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:43:43 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:43:43.346 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:43:43 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:43:43.346 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:43:43 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:43:43.347 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:43:43 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:43:43.347 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:43:43 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:43:43.347 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:43:43 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return future.result(timeout) 2024-03-22T05:43:43.347 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:43:43 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:43:43.347 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:43:43 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.__get_result() 2024-03-22T05:43:43.347 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:43:43 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:43:43.347 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:43:43 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: raise self._exception 2024-03-22T05:43:43.347 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:43:43 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:43:43.347 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:43:43 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: config_blobs=daemon_spec.final_config, 2024-03-22T05:43:43.347 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:43:43 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:43:43.347 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:43:43 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:43:43.347 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:43:43 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:43:43.348 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:43:43 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:43:43.348 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:43:43 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:43:43.348 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:43:43 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:43:43.348 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:43:43 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:43:43.348 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:43:43 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:43:43.348 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:43:43 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:43:43.348 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:43:43 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:43:43.348 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:43:43 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:43:43.348 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:43:43 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:43:43.348 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:43:43 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:43:43.348 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:43:43 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:43:43.348 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:43:43 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:43:43.349 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:43:43 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:43:43.349 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:43:43 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:43:43.349 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:43:43 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:43:43.349 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:43:43 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:43:43.349 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:43:43 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:43:43.349 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:43:43 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:43:43.349 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:43:43 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:43:43.349 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:43:43 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:43:43.605 INFO:teuthology.orchestra.run.smithi005.stdout:true 2024-03-22T05:43:43.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:43 smithi005 ceph-mon[24793]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:43:43.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:43 smithi005 ceph-mon[29523]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:43:44.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:43 smithi186 ceph-mon[25720]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:43:44.290 INFO:teuthology.orchestra.run.smithi005.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T05:43:44.291 INFO:teuthology.orchestra.run.smithi005.stdout:alertmanager.a smithi005 *:9093,9094 running (78s) 73s ago 23m 17.1M - 0.25.0 c8568f914cd2 ee99dfc435e8 2024-03-22T05:43:44.291 INFO:teuthology.orchestra.run.smithi005.stdout:grafana.a smithi186 *:3000 running (62s) 57s ago 22m 49.3M - 9.4.12 2bacad6d85d8 16fd57dbbd71 2024-03-22T05:43:44.291 INFO:teuthology.orchestra.run.smithi005.stdout:iscsi.foo.smithi005.mcdwbp smithi005 unknown 73s ago 22m 58.9M - 3.5 e1d6a67b021e 5215e03bf81b 2024-03-22T05:43:44.291 INFO:teuthology.orchestra.run.smithi005.stdout:mgr.x smithi186 *:8443,9283,8765 running (2m) 57s ago 27m 449M - 19.0.0-2299-gea01217f aa461e7c5e7f c556421a73f3 2024-03-22T05:43:44.291 INFO:teuthology.orchestra.run.smithi005.stdout:mgr.y smithi005 *:8443,9283,8765 running (12m) 73s ago 29m 503M - 19.0.0-2299-gea01217f aa461e7c5e7f 62a813eebc0d 2024-03-22T05:43:44.291 INFO:teuthology.orchestra.run.smithi005.stdout:mon.a smithi005 running (29m) 73s ago 29m 78.5M 2048M 17.2.0 e1d6a67b021e 38e0bba4bb03 2024-03-22T05:43:44.291 INFO:teuthology.orchestra.run.smithi005.stdout:mon.b smithi186 running (27m) 57s ago 27m 53.6M 2048M 17.2.0 e1d6a67b021e 323f53f932f6 2024-03-22T05:43:44.291 INFO:teuthology.orchestra.run.smithi005.stdout:mon.c smithi005 running (28m) 73s ago 28m 58.8M 2048M 17.2.0 e1d6a67b021e 00ec24d38612 2024-03-22T05:43:44.291 INFO:teuthology.orchestra.run.smithi005.stdout:node-exporter.a smithi005 *:9100 running (2m) 73s ago 23m 8786k - 1.5.0 0da6a335fe13 f635cd72b9d3 2024-03-22T05:43:44.291 INFO:teuthology.orchestra.run.smithi005.stdout:node-exporter.b smithi186 *:9100 running (114s) 57s ago 23m 14.5M - 1.5.0 0da6a335fe13 d7e0b3d20191 2024-03-22T05:43:44.291 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0 smithi005 running (27m) 73s ago 27m 53.5M 2993M 17.2.0 e1d6a67b021e ef5c7053dd37 2024-03-22T05:43:44.292 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1 smithi005 running (26m) 73s ago 26m 56.5M 2993M 17.2.0 e1d6a67b021e a1e7ee1017b6 2024-03-22T05:43:44.292 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2 smithi005 running (26m) 73s ago 26m 48.5M 2993M 17.2.0 e1d6a67b021e 462ea63ad0e5 2024-03-22T05:43:44.292 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3 smithi005 running (26m) 73s ago 26m 56.5M 2993M 17.2.0 e1d6a67b021e b25432a7239b 2024-03-22T05:43:44.292 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4 smithi186 running (25m) 57s ago 25m 56.1M 3248M 17.2.0 e1d6a67b021e cd963671f340 2024-03-22T05:43:44.292 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5 smithi186 running (25m) 57s ago 25m 51.6M 3248M 17.2.0 e1d6a67b021e f98c87f1a2cd 2024-03-22T05:43:44.292 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6 smithi186 running (25m) 57s ago 25m 51.7M 3248M 17.2.0 e1d6a67b021e 704ac49c68c4 2024-03-22T05:43:44.292 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7 smithi186 running (24m) 57s ago 24m 54.1M 3248M 17.2.0 e1d6a67b021e f415b90edf63 2024-03-22T05:43:44.292 INFO:teuthology.orchestra.run.smithi005.stdout:prometheus.a smithi186 *:9095 running (95s) 57s ago 23m 60.0M - 2.43.0 a07b618ecd1d a5a043588d6d 2024-03-22T05:43:44.292 INFO:teuthology.orchestra.run.smithi005.stdout:rgw.foo.smithi005.exdhqu smithi005 *:8000 running (22m) 73s ago 22m 78.8M - 17.2.0 e1d6a67b021e 6bb27fe11553 2024-03-22T05:43:44.292 INFO:teuthology.orchestra.run.smithi005.stdout:rgw.foo.smithi186.mcgodk smithi186 *:8000 running (22m) 57s ago 22m 77.2M - 17.2.0 e1d6a67b021e 19b5c20a34f7 2024-03-22T05:43:44.699 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-03-22T05:43:44.699 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": { 2024-03-22T05:43:44.699 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-03-22T05:43:44.699 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:43:44.699 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr": { 2024-03-22T05:43:44.700 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev)": 2 2024-03-22T05:43:44.700 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:43:44.700 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": { 2024-03-22T05:43:44.700 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-22T05:43:44.700 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:43:44.700 INFO:teuthology.orchestra.run.smithi005.stdout: "mds": {}, 2024-03-22T05:43:44.700 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw": { 2024-03-22T05:43:44.700 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-22T05:43:44.700 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:43:44.700 INFO:teuthology.orchestra.run.smithi005.stdout: "overall": { 2024-03-22T05:43:44.700 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-03-22T05:43:44.700 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev)": 2 2024-03-22T05:43:44.700 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-03-22T05:43:44.700 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-03-22T05:43:44.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:44 smithi005 ceph-mon[29523]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:43:44.952 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:43:44.952 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:43:44.952 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:43:44.952 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:43:44.952 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:43:44.952 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:43:44.952 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:43:44.952 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:43:44.953 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:43:44.953 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:43:44.953 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:43:44.953 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:43:44.953 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:43:44.953 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:43:44.953 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:43:44.953 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:43:44.953 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:43:44.953 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:43:44.953 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:43:44.953 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:43:44.953 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:43:44.954 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:43:44.954 INFO:journalctl@ceph.mon.c.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:43:44.954 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:43:44.954 INFO:journalctl@ceph.mon.c.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:43:44.954 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:43:44.954 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:43:44.954 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:43:44.954 INFO:journalctl@ceph.mon.c.smithi005.stdout: return future.result(timeout) 2024-03-22T05:43:44.954 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:43:44.954 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.__get_result() 2024-03-22T05:43:44.954 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:43:44.954 INFO:journalctl@ceph.mon.c.smithi005.stdout: raise self._exception 2024-03-22T05:43:44.954 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:43:44.955 INFO:journalctl@ceph.mon.c.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:43:44.955 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:43:44.955 INFO:journalctl@ceph.mon.c.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:43:44.955 INFO:journalctl@ceph.mon.c.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:43:44.955 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:43:44.955 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:43:44.955 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:43:44.955 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:43:44.955 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:43:44.955 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:43:44.955 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:43:44.955 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:43:44.955 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:43:44.955 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:43:44.955 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:43:44.956 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:43:44.956 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:43:44.956 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:43:44.956 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:43:44.956 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:44 smithi005 ceph-mon[24793]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:43:44.956 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:43:44.957 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:43:44.957 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:43:44.957 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:43:44.958 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:43:44.958 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:43:44.958 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:43:44.958 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:43:44.958 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:43:44.958 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:43:44.958 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:43:44.958 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:43:44.958 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:43:44.958 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:43:44.958 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:43:44.958 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:43:44.958 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:43:44.958 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:43:44.959 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:43:44.959 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:43:44.959 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:43:44.959 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:43:44.959 INFO:journalctl@ceph.mon.a.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:43:44.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:43:44.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:43:44.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:43:44.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:43:44.960 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:43:44.960 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:44 smithi005 ceph-mon[29523]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37 2024-03-22T05:43:44.960 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:44 smithi005 ceph-mon[29523]: from='client.25507 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:43:44.960 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:43:44.960 INFO:journalctl@ceph.mon.a.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:43:44.960 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:43:44.960 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:43:44.960 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:43:44.960 INFO:journalctl@ceph.mon.a.smithi005.stdout: return future.result(timeout) 2024-03-22T05:43:44.961 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:43:44.961 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.__get_result() 2024-03-22T05:43:44.961 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:43:44.961 INFO:journalctl@ceph.mon.a.smithi005.stdout: raise self._exception 2024-03-22T05:43:44.961 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:43:44.961 INFO:journalctl@ceph.mon.a.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:43:44.961 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:43:44.961 INFO:journalctl@ceph.mon.a.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:43:44.961 INFO:journalctl@ceph.mon.a.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:43:44.961 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:43:44.961 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:43:44.961 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:43:44.961 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:43:44.961 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:43:44.961 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:43:44.962 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:43:44.962 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:43:44.962 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:43:44.962 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:43:44.962 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:43:44.962 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:43:44.962 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:43:44.962 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:43:44.962 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:43:44.962 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:43:44.962 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:43:44.962 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:43:44.962 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:43:44.962 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:43:44.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:44 smithi005 ceph-mon[24793]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37 2024-03-22T05:43:44.963 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:44 smithi005 ceph-mon[24793]: from='client.25507 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:43:45.040 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-03-22T05:43:45.040 INFO:teuthology.orchestra.run.smithi005.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37", 2024-03-22T05:43:45.040 INFO:teuthology.orchestra.run.smithi005.stdout: "in_progress": true, 2024-03-22T05:43:45.040 INFO:teuthology.orchestra.run.smithi005.stdout: "which": "Upgrading daemons of type(s) mon on host(s) smithi186", 2024-03-22T05:43:45.040 INFO:teuthology.orchestra.run.smithi005.stdout: "services_complete": [], 2024-03-22T05:43:45.040 INFO:teuthology.orchestra.run.smithi005.stdout: "progress": "", 2024-03-22T05:43:45.041 INFO:teuthology.orchestra.run.smithi005.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37 image", 2024-03-22T05:43:45.041 INFO:teuthology.orchestra.run.smithi005.stdout: "is_paused": false 2024-03-22T05:43:45.041 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-03-22T05:43:45.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:44 smithi186 ceph-mon[25720]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:43:45.043 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:43:45.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:43:45.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:43:45.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:43:45.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:43:45.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:43:45.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:43:45.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:43:45.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:43:45.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:43:45.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:43:45.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:43:45.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:43:45.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:43:45.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:43:45.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:43:45.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:43:45.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:43:45.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:43:45.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:43:45.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:43:45.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:43:45.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:43:45.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:43:45.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:43:45.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:43:45.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:43:45.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:43:45.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: return future.result(timeout) 2024-03-22T05:43:45.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:43:45.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.__get_result() 2024-03-22T05:43:45.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:43:45.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: raise self._exception 2024-03-22T05:43:45.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:43:45.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:43:45.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:43:45.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:43:45.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:43:45.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:43:45.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:43:45.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:43:45.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:43:45.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:43:45.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:43:45.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:43:45.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:43:45.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:43:45.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:43:45.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:43:45.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:43:45.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:43:45.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:43:45.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:43:45.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:43:45.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:43:45.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:43:45.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:43:45.048 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:43:45.048 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:44 smithi186 ceph-mon[25720]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37 2024-03-22T05:43:45.048 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:44 smithi186 ceph-mon[25720]: from='client.25507 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:43:45.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:45 smithi005 ceph-mon[24793]: from='client.25513 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:43:45.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:45 smithi005 ceph-mon[24793]: from='client.15618 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:43:45.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:45 smithi005 ceph-mon[24793]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:43:45.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:45 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/1005856356' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:43:45.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:45 smithi005 ceph-mon[29523]: from='client.25513 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:43:45.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:45 smithi005 ceph-mon[29523]: from='client.15618 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:43:45.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:45 smithi005 ceph-mon[29523]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:43:45.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:45 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/1005856356' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:43:46.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:45 smithi186 ceph-mon[25720]: from='client.25513 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:43:46.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:45 smithi186 ceph-mon[25720]: from='client.15618 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:43:46.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:45 smithi186 ceph-mon[25720]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:43:46.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:45 smithi186 ceph-mon[25720]: from='client.? 172.21.15.5:0/1005856356' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:43:46.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:46 smithi005 ceph-mon[24793]: from='client.25531 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:43:46.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:46 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:43:46.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:46 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:43:46.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:46 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:43:46.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:46 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:43:46.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:46 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-22T05:43:46.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:46 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-03-22T05:43:46.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:46 smithi005 ceph-mon[29523]: from='client.25531 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:43:46.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:46 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:43:46.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:46 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:43:46.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:46 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:43:46.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:46 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:43:46.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:46 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-22T05:43:46.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:46 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-03-22T05:43:47.008 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:46 smithi186 ceph-mon[25720]: from='client.25531 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:43:47.008 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:46 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:43:47.009 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:46 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:43:47.009 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:46 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:43:47.009 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:46 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:43:47.009 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:46 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-22T05:43:47.009 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:46 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-03-22T05:43:47.452 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:43:46 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:43:46] "GET /metrics HTTP/1.1" 200 37988 "" "Prometheus/2.43.0" 2024-03-22T05:43:47.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:47 smithi005 ceph-mon[24793]: Upgrade: Target is version 19.0.0-2299-gea01217f (squid) 2024-03-22T05:43:47.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:47 smithi005 ceph-mon[24793]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, digests ['quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9'] 2024-03-22T05:43:47.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:47 smithi005 ceph-mon[24793]: Upgrade: Setting container_image for all mgr 2024-03-22T05:43:47.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:47 smithi005 ceph-mon[24793]: Upgrade: It appears safe to stop mon.b 2024-03-22T05:43:47.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:47 smithi005 ceph-mon[24793]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:43:47.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:47 smithi005 ceph-mon[29523]: Upgrade: Target is version 19.0.0-2299-gea01217f (squid) 2024-03-22T05:43:47.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:47 smithi005 ceph-mon[29523]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, digests ['quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9'] 2024-03-22T05:43:47.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:47 smithi005 ceph-mon[29523]: Upgrade: Setting container_image for all mgr 2024-03-22T05:43:47.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:47 smithi005 ceph-mon[29523]: Upgrade: It appears safe to stop mon.b 2024-03-22T05:43:47.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:47 smithi005 ceph-mon[29523]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:43:47.997 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:47 smithi186 ceph-mon[25720]: Upgrade: Target is version 19.0.0-2299-gea01217f (squid) 2024-03-22T05:43:47.997 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:47 smithi186 ceph-mon[25720]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, digests ['quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9'] 2024-03-22T05:43:47.997 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:47 smithi186 ceph-mon[25720]: Upgrade: Setting container_image for all mgr 2024-03-22T05:43:47.997 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:47 smithi186 ceph-mon[25720]: Upgrade: It appears safe to stop mon.b 2024-03-22T05:43:47.997 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:47 smithi186 ceph-mon[25720]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:43:49.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:49 smithi186 ceph-mon[25720]: Upgrade: Updating mon.b 2024-03-22T05:43:49.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:49 smithi186 ceph-mon[25720]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:43:49.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:49 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-22T05:43:49.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:49 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-22T05:43:49.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:49 smithi186 ceph-mon[25720]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:43:49.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:49 smithi186 ceph-mon[25720]: Deploying daemon mon.b on smithi186 2024-03-22T05:43:49.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:49 smithi186 ceph-mon[25720]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 554 B/s rd, 0 op/s 2024-03-22T05:43:49.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:49 smithi005 ceph-mon[24793]: Upgrade: Updating mon.b 2024-03-22T05:43:49.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:49 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:43:49.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:49 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-22T05:43:49.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:49 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-22T05:43:49.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:49 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:43:49.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:49 smithi005 ceph-mon[24793]: Deploying daemon mon.b on smithi186 2024-03-22T05:43:49.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:49 smithi005 ceph-mon[24793]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 554 B/s rd, 0 op/s 2024-03-22T05:43:49.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:49 smithi005 ceph-mon[29523]: Upgrade: Updating mon.b 2024-03-22T05:43:49.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:49 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:43:49.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:49 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-22T05:43:49.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:49 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-22T05:43:49.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:49 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:43:49.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:49 smithi005 ceph-mon[29523]: Deploying daemon mon.b on smithi186 2024-03-22T05:43:49.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:49 smithi005 ceph-mon[29523]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 554 B/s rd, 0 op/s 2024-03-22T05:43:49.959 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:49 smithi186 systemd[1]: Stopping Ceph mon.b for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:43:50.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mon-b[25716]: 2024-03-22T05:43:49.957+0000 7f2c41955700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T05:43:50.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mon-b[25716]: 2024-03-22T05:43:49.957+0000 7f2c41955700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-03-22T05:43:50.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:50 smithi186 podman[67740]: 2024-03-22 05:43:50.081174407 +0000 UTC m=+0.243405048 container died 323f53f932f651b46684b8054199cd7093279274961dcca1ae6995158bfb3583 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mon-b, maintainer=Guillaume Abrioux , GIT_CLEAN=True, name=centos-stream, ceph=True, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, distribution-scope=public, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870) 2024-03-22T05:43:50.633 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:50 smithi186 podman[67740]: 2024-03-22 05:43:50.39999971 +0000 UTC m=+0.562230351 container cleanup 323f53f932f651b46684b8054199cd7093279274961dcca1ae6995158bfb3583 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mon-b, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, architecture=x86_64, vcs-type=git, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, distribution-scope=public, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, version=8, build-date=2022-05-03T08:36:31.336870) 2024-03-22T05:43:50.633 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:50 smithi186 bash[67740]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mon-b 2024-03-22T05:43:51.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:50 smithi186 podman[67754]: 2024-03-22 05:43:50.634644499 +0000 UTC m=+0.556452451 container remove 323f53f932f651b46684b8054199cd7093279274961dcca1ae6995158bfb3583 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mon-b, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, architecture=x86_64, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., io.openshift.expose-services=, io.buildah.version=1.19.8) 2024-03-22T05:43:51.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:50 smithi186 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@mon.b.service: Deactivated successfully. 2024-03-22T05:43:51.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:50 smithi186 systemd[1]: Stopped Ceph mon.b for f59f36e2-e80a-11ee-95cd-87774f69a715. 2024-03-22T05:43:51.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:50 smithi186 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@mon.b.service: Consumed 17.839s CPU time. 2024-03-22T05:43:51.399 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 systemd[1]: Starting Ceph mon.b for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:43:51.399 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 podman[67884]: 2024-03-22 05:43:51.172432205 +0000 UTC m=+0.018098327 image pull aa461e7c5e7f322b169c789b389d594b994ea7ffcbc9e24e563a3ec9b69bc3c1 quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9 2024-03-22T05:43:51.399 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 podman[67884]: 2024-03-22 05:43:51.277729675 +0000 UTC m=+0.123395831 container create c1798feac15b559b751d8adbbc6ff346ae094c5e47fdd03cbfa15609427fac0b (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mon-b, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-22T05:43:51.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 podman[67884]: 2024-03-22 05:43:51.514117265 +0000 UTC m=+0.359783387 container init c1798feac15b559b751d8adbbc6ff346ae094c5e47fdd03cbfa15609427fac0b (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mon-b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-22T05:43:51.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 podman[67884]: 2024-03-22 05:43:51.517428 +0000 UTC m=+0.363094119 container start c1798feac15b559b751d8adbbc6ff346ae094c5e47fdd03cbfa15609427fac0b (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mon-b, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux ) 2024-03-22T05:43:51.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: set uid:gid to 167:167 (ceph:ceph) 2024-03-22T05:43:51.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev), process ceph-mon, pid 2 2024-03-22T05:43:51.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: pidfile_write: ignore empty --pid-file 2024-03-22T05:43:51.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: load: jerasure load: lrc 2024-03-22T05:43:51.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: RocksDB version: 7.9.2 2024-03-22T05:43:51.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Git sha 0 2024-03-22T05:43:51.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Compile date 2024-03-21 23:09:57 2024-03-22T05:43:51.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: DB SUMMARY 2024-03-22T05:43:51.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: DB Session ID: 67YC3H9V6N5V3LM6B2GN 2024-03-22T05:43:51.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: CURRENT file: CURRENT 2024-03-22T05:43:51.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: IDENTITY file: IDENTITY 2024-03-22T05:43:51.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: MANIFEST file: MANIFEST-000009 size: 2814 Bytes 2024-03-22T05:43:51.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 1, files: 000054.sst 2024-03-22T05:43:51.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000052.log size: 2204027 ; 2024-03-22T05:43:51.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.error_if_exists: 0 2024-03-22T05:43:51.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.create_if_missing: 0 2024-03-22T05:43:51.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.paranoid_checks: 1 2024-03-22T05:43:51.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-22T05:43:51.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-22T05:43:51.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-22T05:43:51.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.env: 0x55f61b5e1b20 2024-03-22T05:43:51.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.fs: PosixFileSystem 2024-03-22T05:43:51.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.info_log: 0x55f61c417920 2024-03-22T05:43:51.796 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.max_file_opening_threads: 16 2024-03-22T05:43:51.797 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.statistics: (nil) 2024-03-22T05:43:51.797 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.use_fsync: 0 2024-03-22T05:43:51.797 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.max_log_file_size: 0 2024-03-22T05:43:51.797 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-22T05:43:51.797 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-22T05:43:51.797 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.keep_log_file_num: 1000 2024-03-22T05:43:51.797 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.recycle_log_file_num: 0 2024-03-22T05:43:51.797 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.allow_fallocate: 1 2024-03-22T05:43:51.797 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.allow_mmap_reads: 0 2024-03-22T05:43:51.797 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.allow_mmap_writes: 0 2024-03-22T05:43:51.798 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.use_direct_reads: 0 2024-03-22T05:43:51.798 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-22T05:43:51.798 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.create_missing_column_families: 0 2024-03-22T05:43:51.798 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.db_log_dir: 2024-03-22T05:43:51.798 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.wal_dir: 2024-03-22T05:43:51.798 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-22T05:43:51.798 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-22T05:43:51.798 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-22T05:43:51.798 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-22T05:43:51.798 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-22T05:43:51.798 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-22T05:43:51.798 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.advise_random_on_open: 1 2024-03-22T05:43:51.798 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.db_write_buffer_size: 0 2024-03-22T05:43:51.798 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.write_buffer_manager: 0x55f61c3e81e0 2024-03-22T05:43:51.799 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-22T05:43:51.799 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-22T05:43:51.799 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-22T05:43:51.799 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.rate_limiter: (nil) 2024-03-22T05:43:51.799 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-22T05:43:51.799 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.wal_recovery_mode: 2 2024-03-22T05:43:51.799 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.enable_thread_tracking: 0 2024-03-22T05:43:51.799 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.enable_pipelined_write: 0 2024-03-22T05:43:51.799 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.unordered_write: 0 2024-03-22T05:43:51.799 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-22T05:43:51.799 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-22T05:43:51.799 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-22T05:43:51.799 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-22T05:43:51.799 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.row_cache: None 2024-03-22T05:43:51.799 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.wal_filter: None 2024-03-22T05:43:51.799 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-22T05:43:51.800 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.allow_ingest_behind: 0 2024-03-22T05:43:51.800 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.two_write_queues: 0 2024-03-22T05:43:51.800 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.manual_wal_flush: 0 2024-03-22T05:43:51.800 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.wal_compression: 0 2024-03-22T05:43:51.800 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.atomic_flush: 0 2024-03-22T05:43:51.800 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-22T05:43:51.800 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-22T05:43:51.800 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-22T05:43:51.800 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.log_readahead_size: 0 2024-03-22T05:43:51.800 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-22T05:43:51.800 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.best_efforts_recovery: 0 2024-03-22T05:43:51.800 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-22T05:43:51.800 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-22T05:43:51.801 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.allow_data_in_errors: 0 2024-03-22T05:43:51.801 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.db_host_id: __hostname__ 2024-03-22T05:43:51.801 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-22T05:43:51.801 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.max_background_jobs: 2 2024-03-22T05:43:51.801 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.max_background_compactions: -1 2024-03-22T05:43:51.801 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.max_subcompactions: 1 2024-03-22T05:43:51.801 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-22T05:43:51.801 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-22T05:43:51.801 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-22T05:43:51.801 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.max_total_wal_size: 0 2024-03-22T05:43:51.801 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-22T05:43:51.801 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-22T05:43:51.801 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-22T05:43:51.801 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-22T05:43:51.801 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.max_open_files: -1 2024-03-22T05:43:51.802 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.bytes_per_sync: 0 2024-03-22T05:43:51.802 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-22T05:43:51.802 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-22T05:43:51.802 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.compaction_readahead_size: 0 2024-03-22T05:43:51.802 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.max_background_flushes: -1 2024-03-22T05:43:51.802 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Compression algorithms supported: 2024-03-22T05:43:51.802 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-22T05:43:51.802 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: kZSTD supported: 0 2024-03-22T05:43:51.802 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: kXpressCompression supported: 0 2024-03-22T05:43:51.802 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: kLZ4HCCompression supported: 1 2024-03-22T05:43:51.802 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: kZlibCompression supported: 1 2024-03-22T05:43:51.802 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: kSnappyCompression supported: 1 2024-03-22T05:43:51.802 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: kLZ4Compression supported: 1 2024-03-22T05:43:51.802 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: kBZip2Compression supported: 0 2024-03-22T05:43:51.803 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-22T05:43:51.803 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-22T05:43:51.803 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000009 2024-03-22T05:43:51.803 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-22T05:43:51.803 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-22T05:43:51.803 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.merge_operator: 2024-03-22T05:43:51.803 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.compaction_filter: None 2024-03-22T05:43:51.803 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.compaction_filter_factory: None 2024-03-22T05:43:51.803 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.sst_partitioner_factory: None 2024-03-22T05:43:51.803 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-22T05:43:51.803 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-22T05:43:51.803 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55f61c417a20) 2024-03-22T05:43:51.803 INFO:journalctl@ceph.mon.b.smithi186.stdout: cache_index_and_filter_blocks: 1 2024-03-22T05:43:51.803 INFO:journalctl@ceph.mon.b.smithi186.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-22T05:43:51.803 INFO:journalctl@ceph.mon.b.smithi186.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-22T05:43:51.804 INFO:journalctl@ceph.mon.b.smithi186.stdout: pin_top_level_index_and_filter: 1 2024-03-22T05:43:51.804 INFO:journalctl@ceph.mon.b.smithi186.stdout: index_type: 0 2024-03-22T05:43:51.804 INFO:journalctl@ceph.mon.b.smithi186.stdout: data_block_index_type: 0 2024-03-22T05:43:51.804 INFO:journalctl@ceph.mon.b.smithi186.stdout: index_shortening: 1 2024-03-22T05:43:51.804 INFO:journalctl@ceph.mon.b.smithi186.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-22T05:43:51.804 INFO:journalctl@ceph.mon.b.smithi186.stdout: checksum: 4 2024-03-22T05:43:51.804 INFO:journalctl@ceph.mon.b.smithi186.stdout: no_block_cache: 0 2024-03-22T05:43:51.804 INFO:journalctl@ceph.mon.b.smithi186.stdout: block_cache: 0x55f61c41b090 2024-03-22T05:43:51.804 INFO:journalctl@ceph.mon.b.smithi186.stdout: block_cache_name: BinnedLRUCache 2024-03-22T05:43:51.804 INFO:journalctl@ceph.mon.b.smithi186.stdout: block_cache_options: 2024-03-22T05:43:51.804 INFO:journalctl@ceph.mon.b.smithi186.stdout: capacity : 536870912 2024-03-22T05:43:51.804 INFO:journalctl@ceph.mon.b.smithi186.stdout: num_shard_bits : 4 2024-03-22T05:43:51.804 INFO:journalctl@ceph.mon.b.smithi186.stdout: strict_capacity_limit : 0 2024-03-22T05:43:51.804 INFO:journalctl@ceph.mon.b.smithi186.stdout: high_pri_pool_ratio: 0.000 2024-03-22T05:43:51.805 INFO:journalctl@ceph.mon.b.smithi186.stdout: block_cache_compressed: (nil) 2024-03-22T05:43:51.805 INFO:journalctl@ceph.mon.b.smithi186.stdout: persistent_cache: (nil) 2024-03-22T05:43:51.805 INFO:journalctl@ceph.mon.b.smithi186.stdout: block_size: 4096 2024-03-22T05:43:51.805 INFO:journalctl@ceph.mon.b.smithi186.stdout: block_size_deviation: 10 2024-03-22T05:43:51.805 INFO:journalctl@ceph.mon.b.smithi186.stdout: block_restart_interval: 16 2024-03-22T05:43:51.805 INFO:journalctl@ceph.mon.b.smithi186.stdout: index_block_restart_interval: 1 2024-03-22T05:43:51.805 INFO:journalctl@ceph.mon.b.smithi186.stdout: metadata_block_size: 4096 2024-03-22T05:43:51.805 INFO:journalctl@ceph.mon.b.smithi186.stdout: partition_filters: 0 2024-03-22T05:43:51.805 INFO:journalctl@ceph.mon.b.smithi186.stdout: use_delta_encoding: 1 2024-03-22T05:43:51.805 INFO:journalctl@ceph.mon.b.smithi186.stdout: filter_policy: bloomfilter 2024-03-22T05:43:51.805 INFO:journalctl@ceph.mon.b.smithi186.stdout: whole_key_filtering: 1 2024-03-22T05:43:51.805 INFO:journalctl@ceph.mon.b.smithi186.stdout: verify_compression: 0 2024-03-22T05:43:51.805 INFO:journalctl@ceph.mon.b.smithi186.stdout: read_amp_bytes_per_bit: 0 2024-03-22T05:43:51.805 INFO:journalctl@ceph.mon.b.smithi186.stdout: format_version: 5 2024-03-22T05:43:51.805 INFO:journalctl@ceph.mon.b.smithi186.stdout: enable_index_compression: 1 2024-03-22T05:43:51.806 INFO:journalctl@ceph.mon.b.smithi186.stdout: block_align: 0 2024-03-22T05:43:51.806 INFO:journalctl@ceph.mon.b.smithi186.stdout: max_auto_readahead_size: 262144 2024-03-22T05:43:51.806 INFO:journalctl@ceph.mon.b.smithi186.stdout: prepopulate_block_cache: 0 2024-03-22T05:43:51.806 INFO:journalctl@ceph.mon.b.smithi186.stdout: initial_auto_readahead_size: 8192 2024-03-22T05:43:51.806 INFO:journalctl@ceph.mon.b.smithi186.stdout: num_file_reads_for_auto_readahead: 2 2024-03-22T05:43:51.806 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.write_buffer_size: 33554432 2024-03-22T05:43:51.806 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.max_write_buffer_number: 2 2024-03-22T05:43:51.806 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.compression: NoCompression 2024-03-22T05:43:51.806 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.bottommost_compression: Disabled 2024-03-22T05:43:51.806 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.prefix_extractor: nullptr 2024-03-22T05:43:51.806 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-22T05:43:51.806 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.num_levels: 7 2024-03-22T05:43:51.806 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-22T05:43:51.806 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-22T05:43:51.807 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-22T05:43:51.807 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-22T05:43:51.807 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-22T05:43:51.807 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-22T05:43:51.807 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-22T05:43:51.807 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-22T05:43:51.807 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-22T05:43:51.807 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-22T05:43:51.807 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-22T05:43:51.807 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-22T05:43:51.807 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-22T05:43:51.807 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.compression_opts.level: 32767 2024-03-22T05:43:51.807 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.compression_opts.strategy: 0 2024-03-22T05:43:51.807 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-22T05:43:51.807 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-22T05:43:51.807 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-22T05:43:51.808 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-22T05:43:51.808 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.compression_opts.enabled: false 2024-03-22T05:43:51.808 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-22T05:43:51.808 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-22T05:43:51.808 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-22T05:43:51.808 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-22T05:43:51.808 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.target_file_size_base: 67108864 2024-03-22T05:43:51.808 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-22T05:43:51.808 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-22T05:43:51.808 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-22T05:43:51.808 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-22T05:43:51.808 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-22T05:43:51.808 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-22T05:43:51.808 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-22T05:43:51.809 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-22T05:43:51.809 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-22T05:43:51.809 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-22T05:43:51.809 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-22T05:43:51.809 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-22T05:43:51.809 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-22T05:43:51.809 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-22T05:43:51.809 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.arena_block_size: 1048576 2024-03-22T05:43:51.809 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-22T05:43:51.809 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-22T05:43:51.809 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.disable_auto_compactions: 0 2024-03-22T05:43:51.809 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-22T05:43:51.809 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-22T05:43:51.809 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-22T05:43:51.809 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-22T05:43:51.810 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-22T05:43:51.810 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-22T05:43:51.810 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-22T05:43:51.810 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-22T05:43:51.810 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-22T05:43:51.810 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-22T05:43:51.810 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.table_properties_collectors: 2024-03-22T05:43:51.810 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.inplace_update_support: 0 2024-03-22T05:43:51.810 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-22T05:43:51.810 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-22T05:43:51.810 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-22T05:43:51.810 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-22T05:43:51.810 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.bloom_locality: 0 2024-03-22T05:43:51.810 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.max_successive_merges: 0 2024-03-22T05:43:51.811 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-22T05:43:51.811 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.paranoid_file_checks: 0 2024-03-22T05:43:51.811 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.force_consistency_checks: 1 2024-03-22T05:43:51.811 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.report_bg_io_stats: 0 2024-03-22T05:43:51.811 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.ttl: 2592000 2024-03-22T05:43:51.811 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-22T05:43:51.811 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-22T05:43:51.811 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-22T05:43:51.811 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.enable_blob_files: false 2024-03-22T05:43:51.811 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.min_blob_size: 0 2024-03-22T05:43:51.811 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.blob_file_size: 268435456 2024-03-22T05:43:51.811 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-22T05:43:51.811 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-22T05:43:51.812 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-22T05:43:51.812 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-22T05:43:51.812 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-22T05:43:51.812 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.blob_file_starting_level: 0 2024-03-22T05:43:51.812 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-22T05:43:51.812 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 54.sst 2024-03-22T05:43:51.812 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-22T05:43:51.812 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 56, last_sequence is 28910, log_number is 52,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-22T05:43:51.812 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 52 2024-03-22T05:43:51.812 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 9e536753-d7a3-4f36-9e19-d49e4b081359 2024-03-22T05:43:51.812 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711086231547427, "job": 1, "event": "recovery_started", "wal_files": [52]} 2024-03-22T05:43:51.812 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #52 mode 2 2024-03-22T05:43:51.812 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711086231555631, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 57, "file_size": 1404985, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 28915, "largest_seqno": 30259, "table_properties": {"data_size": 1399338, "index_size": 3224, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1413, "raw_key_size": 13618, "raw_average_key_size": 24, "raw_value_size": 1387798, "raw_average_value_size": 2523, "num_data_blocks": 143, "num_entries": 550, "num_filter_entries": 550, "num_deletions": 7, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711086231, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "9e536753-d7a3-4f36-9e19-d49e4b081359", "db_session_id": "67YC3H9V6N5V3LM6B2GN", "orig_file_number": 57, "seqno_to_time_mapping": "N/A"}} 2024-03-22T05:43:51.812 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711086231555739, "job": 1, "event": "recovery_finished"} 2024-03-22T05:43:51.812 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: [db/version_set.cc:5047] Creating manifest 59 2024-03-22T05:43:51.813 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-22T05:43:51.813 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000052.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-22T05:43:51.813 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55f61c50a000 2024-03-22T05:43:51.813 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: DB pointer 0x55f61c4f4000 2024-03-22T05:43:51.813 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-22T05:43:51.813 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-22T05:43:51.813 INFO:journalctl@ceph.mon.b.smithi186.stdout: ** DB Stats ** 2024-03-22T05:43:51.813 INFO:journalctl@ceph.mon.b.smithi186.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-22T05:43:51.813 INFO:journalctl@ceph.mon.b.smithi186.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-22T05:43:51.813 INFO:journalctl@ceph.mon.b.smithi186.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-22T05:43:51.813 INFO:journalctl@ceph.mon.b.smithi186.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-22T05:43:51.813 INFO:journalctl@ceph.mon.b.smithi186.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-22T05:43:51.813 INFO:journalctl@ceph.mon.b.smithi186.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-22T05:43:51.814 INFO:journalctl@ceph.mon.b.smithi186.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-22T05:43:51.814 INFO:journalctl@ceph.mon.b.smithi186.stdout: 2024-03-22T05:43:51.814 INFO:journalctl@ceph.mon.b.smithi186.stdout: ** Compaction Stats [default] ** 2024-03-22T05:43:51.814 INFO:journalctl@ceph.mon.b.smithi186.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-22T05:43:51.814 INFO:journalctl@ceph.mon.b.smithi186.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-22T05:43:51.814 INFO:journalctl@ceph.mon.b.smithi186.stdout: L0 1/0 1.34 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 471.1 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-03-22T05:43:51.814 INFO:journalctl@ceph.mon.b.smithi186.stdout: L6 1/0 8.91 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-03-22T05:43:51.814 INFO:journalctl@ceph.mon.b.smithi186.stdout: Sum 2/0 10.25 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 471.1 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-03-22T05:43:51.814 INFO:journalctl@ceph.mon.b.smithi186.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 471.1 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-03-22T05:43:51.814 INFO:journalctl@ceph.mon.b.smithi186.stdout: 2024-03-22T05:43:51.814 INFO:journalctl@ceph.mon.b.smithi186.stdout: ** Compaction Stats [default] ** 2024-03-22T05:43:51.814 INFO:journalctl@ceph.mon.b.smithi186.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-22T05:43:51.814 INFO:journalctl@ceph.mon.b.smithi186.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-22T05:43:51.814 INFO:journalctl@ceph.mon.b.smithi186.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 471.1 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-03-22T05:43:51.815 INFO:journalctl@ceph.mon.b.smithi186.stdout: 2024-03-22T05:43:51.815 INFO:journalctl@ceph.mon.b.smithi186.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-22T05:43:51.815 INFO:journalctl@ceph.mon.b.smithi186.stdout: 2024-03-22T05:43:51.815 INFO:journalctl@ceph.mon.b.smithi186.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-22T05:43:51.815 INFO:journalctl@ceph.mon.b.smithi186.stdout: Flush(GB): cumulative 0.001, interval 0.001 2024-03-22T05:43:51.815 INFO:journalctl@ceph.mon.b.smithi186.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-22T05:43:51.815 INFO:journalctl@ceph.mon.b.smithi186.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-22T05:43:51.815 INFO:journalctl@ceph.mon.b.smithi186.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-22T05:43:51.815 INFO:journalctl@ceph.mon.b.smithi186.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-22T05:43:51.815 INFO:journalctl@ceph.mon.b.smithi186.stdout: Cumulative compaction: 0.00 GB write, 124.68 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-22T05:43:51.815 INFO:journalctl@ceph.mon.b.smithi186.stdout: Interval compaction: 0.00 GB write, 124.68 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-22T05:43:51.816 INFO:journalctl@ceph.mon.b.smithi186.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-22T05:43:51.816 INFO:journalctl@ceph.mon.b.smithi186.stdout: Block cache BinnedLRUCache@0x55f61c41b090#2 capacity: 512.00 MB usage: 4.86 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.6e-05 secs_since: 0 2024-03-22T05:43:51.816 INFO:journalctl@ceph.mon.b.smithi186.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,1.53 KB,0.000292063%) IndexBlock(1,3.33 KB,0.000634789%) Misc(1,0.00 KB,0%) 2024-03-22T05:43:51.816 INFO:journalctl@ceph.mon.b.smithi186.stdout: 2024-03-22T05:43:51.816 INFO:journalctl@ceph.mon.b.smithi186.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-22T05:43:51.816 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: starting mon.b rank 2 at public addrs [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] at bind addrs [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid f59f36e2-e80a-11ee-95cd-87774f69a715 2024-03-22T05:43:51.816 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: mon.b@-1(???) e3 preinit fsid f59f36e2-e80a-11ee-95cd-87774f69a715 2024-03-22T05:43:51.816 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: mon.b@-1(???).mds e1 new map 2024-03-22T05:43:51.816 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: mon.b@-1(???).mds e1 print_map 2024-03-22T05:43:51.816 INFO:journalctl@ceph.mon.b.smithi186.stdout: e1 2024-03-22T05:43:51.816 INFO:journalctl@ceph.mon.b.smithi186.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-22T05:43:51.816 INFO:journalctl@ceph.mon.b.smithi186.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T05:43:51.816 INFO:journalctl@ceph.mon.b.smithi186.stdout: legacy client fscid: -1 2024-03-22T05:43:51.816 INFO:journalctl@ceph.mon.b.smithi186.stdout: 2024-03-22T05:43:51.817 INFO:journalctl@ceph.mon.b.smithi186.stdout: No filesystems configured 2024-03-22T05:43:51.817 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: mon.b@-1(???).osd e104 crush map has features 3314933000854323200, adjusting msgr requires 2024-03-22T05:43:51.817 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: mon.b@-1(???).osd e104 crush map has features 432629239337189376, adjusting msgr requires 2024-03-22T05:43:51.817 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: mon.b@-1(???).osd e104 crush map has features 432629239337189376, adjusting msgr requires 2024-03-22T05:43:51.817 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: mon.b@-1(???).osd e104 crush map has features 432629239337189376, adjusting msgr requires 2024-03-22T05:43:51.817 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 ceph-mon[67903]: mon.b@-1(???).paxosservice(auth 1..26) refresh upgraded, format 0 -> 3 2024-03-22T05:43:51.817 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 bash[67884]: c1798feac15b559b751d8adbbc6ff346ae094c5e47fdd03cbfa15609427fac0b 2024-03-22T05:43:51.817 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:51 smithi186 systemd[1]: Started Ceph mon.b for f59f36e2-e80a-11ee-95cd-87774f69a715. 2024-03-22T05:43:53.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:52 smithi186 ceph-mon[67903]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:43:53.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:52 smithi186 ceph-mon[67903]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:43:53.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:52 smithi186 ceph-mon[67903]: mon.b calling monitor election 2024-03-22T05:43:53.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:52 smithi186 ceph-mon[67903]: mon.a calling monitor election 2024-03-22T05:43:53.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:52 smithi186 ceph-mon[67903]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-22T05:43:53.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:52 smithi186 ceph-mon[67903]: monmap e3: 3 mons at {a=[v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0],b=[v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0],c=[v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0]} 2024-03-22T05:43:53.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:52 smithi186 ceph-mon[67903]: fsmap 2024-03-22T05:43:53.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:52 smithi186 ceph-mon[67903]: osdmap e104: 8 total, 8 up, 8 in 2024-03-22T05:43:53.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:52 smithi186 ceph-mon[67903]: mgrmap e43: y(active, since 2m), standbys: x 2024-03-22T05:43:53.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:52 smithi186 ceph-mon[67903]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-22T05:43:53.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:52 smithi186 ceph-mon[67903]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-22T05:43:53.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:52 smithi186 ceph-mon[67903]: daemon iscsi.foo.smithi005.mcdwbp on smithi005 is in unknown state 2024-03-22T05:43:53.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:52 smithi186 ceph-mon[67903]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:43:53.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:52 smithi186 ceph-mon[67903]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:43:53.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:52 smithi186 ceph-mon[67903]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:43:53.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:52 smithi186 ceph-mon[67903]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:43:53.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:52 smithi005 ceph-mon[24793]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:43:53.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:52 smithi005 ceph-mon[24793]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:43:53.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:52 smithi005 ceph-mon[24793]: mon.b calling monitor election 2024-03-22T05:43:53.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:52 smithi005 ceph-mon[24793]: mon.a calling monitor election 2024-03-22T05:43:53.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:52 smithi005 ceph-mon[24793]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-22T05:43:53.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:52 smithi005 ceph-mon[24793]: monmap e3: 3 mons at {a=[v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0],b=[v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0],c=[v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0]} 2024-03-22T05:43:53.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:52 smithi005 ceph-mon[24793]: fsmap 2024-03-22T05:43:53.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:52 smithi005 ceph-mon[24793]: osdmap e104: 8 total, 8 up, 8 in 2024-03-22T05:43:53.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:52 smithi005 ceph-mon[24793]: mgrmap e43: y(active, since 2m), standbys: x 2024-03-22T05:43:53.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:52 smithi005 ceph-mon[24793]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-22T05:43:53.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:52 smithi005 ceph-mon[24793]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-22T05:43:53.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:52 smithi005 ceph-mon[24793]: daemon iscsi.foo.smithi005.mcdwbp on smithi005 is in unknown state 2024-03-22T05:43:53.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:52 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:43:53.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:52 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:43:53.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:52 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:43:53.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:52 smithi005 ceph-mon[24793]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:43:53.204 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:52 smithi005 ceph-mon[29523]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:43:53.204 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:52 smithi005 ceph-mon[29523]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:43:53.204 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:52 smithi005 ceph-mon[29523]: mon.b calling monitor election 2024-03-22T05:43:53.204 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:52 smithi005 ceph-mon[29523]: mon.a calling monitor election 2024-03-22T05:43:53.204 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:52 smithi005 ceph-mon[29523]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-22T05:43:53.204 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:52 smithi005 ceph-mon[29523]: monmap e3: 3 mons at {a=[v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0],b=[v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0],c=[v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0]} 2024-03-22T05:43:53.204 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:52 smithi005 ceph-mon[29523]: fsmap 2024-03-22T05:43:53.204 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:52 smithi005 ceph-mon[29523]: osdmap e104: 8 total, 8 up, 8 in 2024-03-22T05:43:53.204 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:52 smithi005 ceph-mon[29523]: mgrmap e43: y(active, since 2m), standbys: x 2024-03-22T05:43:53.204 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:52 smithi005 ceph-mon[29523]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-22T05:43:53.204 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:52 smithi005 ceph-mon[29523]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-22T05:43:53.204 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:52 smithi005 ceph-mon[29523]: daemon iscsi.foo.smithi005.mcdwbp on smithi005 is in unknown state 2024-03-22T05:43:53.204 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:52 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:43:53.204 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:52 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:43:53.205 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:52 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:43:53.205 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:52 smithi005 ceph-mon[29523]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:43:54.046 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:53 smithi186 ceph-mon[67903]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:43:54.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:53 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:43:54.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:53 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:43:55.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:54 smithi186 ceph-mon[67903]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:43:55.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:54 smithi005 ceph-mon[29523]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:43:55.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:54 smithi005 ceph-mon[24793]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:43:56.817 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:43:56 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:43:56.563+0000 7f487f569700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-22T05:43:57.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:56 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:43:57.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:56 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:43:57.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:56 smithi005 ceph-mon[24793]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:43:57.202 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:43:56 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:43:56] "GET /metrics HTTP/1.1" 200 37931 "" "Prometheus/2.43.0" 2024-03-22T05:43:57.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:56 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:43:57.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:56 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:43:57.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:56 smithi005 ceph-mon[29523]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:43:57.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:56 smithi186 ceph-mon[67903]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:43:57.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:56 smithi186 ceph-mon[67903]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:43:57.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:56 smithi186 ceph-mon[67903]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:43:58.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:57 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:43:58.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:57 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:43:58.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:57 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:43:58.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:57 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:43:58.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:57 smithi186 ceph-mon[67903]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:43:58.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:57 smithi186 ceph-mon[67903]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:43:59.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:43:59 smithi005 ceph-mon[24793]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:43:59.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:43:59 smithi005 ceph-mon[29523]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:43:59.762 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:43:59 smithi186 ceph-mon[67903]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:44:01.217 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:01 smithi005 ceph-mon[29523]: Detected new or changed devices on smithi186 2024-03-22T05:44:01.217 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:01 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:44:01.217 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:01 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:44:01.217 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:01 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:44:01.217 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:01 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:44:01.217 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:01 smithi005 ceph-mon[29523]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-22T05:44:01.217 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:01 smithi005 ceph-mon[29523]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-22T05:44:01.217 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:01 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:44:01.217 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:01 smithi005 ceph-mon[29523]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:44:01.218 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:01 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:44:01.218 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:01 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:44:01.218 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:01 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:44:01.218 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:01 smithi005 ceph-mon[29523]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:44:01.218 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:01 smithi005 ceph-mon[24793]: Detected new or changed devices on smithi186 2024-03-22T05:44:01.218 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:01 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:44:01.218 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:01 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:44:01.218 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:01 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:44:01.218 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:01 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:44:01.218 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:01 smithi005 ceph-mon[24793]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-22T05:44:01.218 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:01 smithi005 ceph-mon[24793]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-22T05:44:01.219 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:01 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:44:01.219 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:01 smithi005 ceph-mon[24793]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:44:01.219 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:01 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:44:01.219 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:01 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:44:01.219 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:01 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:44:01.219 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:01 smithi005 ceph-mon[24793]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:44:01.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:01 smithi186 ceph-mon[67903]: Detected new or changed devices on smithi186 2024-03-22T05:44:01.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:01 smithi186 ceph-mon[67903]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:44:01.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:01 smithi186 ceph-mon[67903]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:44:01.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:01 smithi186 ceph-mon[67903]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:44:01.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:01 smithi186 ceph-mon[67903]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:44:01.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:01 smithi186 ceph-mon[67903]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-22T05:44:01.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:01 smithi186 ceph-mon[67903]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-22T05:44:01.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:01 smithi186 ceph-mon[67903]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:44:01.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:01 smithi186 ceph-mon[67903]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:44:01.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:01 smithi186 ceph-mon[67903]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:44:01.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:01 smithi186 ceph-mon[67903]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:44:01.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:01 smithi186 ceph-mon[67903]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:44:01.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:01 smithi186 ceph-mon[67903]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:44:02.202 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:44:02.065+0000 7f487c563700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:44:02.202 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:44:02.203 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:44:02.203 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:44:02.203 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:44:02.203 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:44:02.203 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:44:02.203 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:44:02.203 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:44:02.203 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:44:02.203 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:44:02.203 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:44:02.203 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:44:02.203 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:44:02.203 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:44:02.203 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:44:02.203 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:44:02.204 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:44:02.204 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:44:02.204 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:44:02.204 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:44:02.204 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:44:02.204 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:44:02.204 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:44:02.204 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:44:02.205 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:44:02.205 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:44:02.205 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:44:02.205 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:44:02.205 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return future.result(timeout) 2024-03-22T05:44:02.205 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:44:02.205 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.__get_result() 2024-03-22T05:44:02.205 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:44:02.205 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: raise self._exception 2024-03-22T05:44:02.205 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:44:02.205 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: config_blobs=daemon_spec.final_config, 2024-03-22T05:44:02.205 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:44:02.205 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:44:02.205 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:44:02.205 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:44:02.205 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:44:02.206 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:44:02.206 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:44:02.206 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:44:02.206 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:44:02.206 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:44:02.206 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:44:02.206 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:44:02.206 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:44:02.206 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:44:02.206 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:44:02.206 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:44:02.206 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:44:02.206 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:44:02.206 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:44:02.207 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:44:02.207 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:44:02.207 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:44:02.207 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:44:02.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:44:02.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:44:02.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:44:02.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:44:02.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:44:02.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:44:02.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:44:02.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:44:02.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:44:02.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:44:02.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:44:02.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-22T05:44:02.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-22T05:44:02.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-03-22T05:44:02.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:44:02.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:44:02.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:44:02.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:44:02.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:44:02.546 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:44:02.546 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:44:02.546 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-22T05:44:02.546 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-22T05:44:02.546 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:44:02.546 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-22T05:44:02.546 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-22T05:44:02.546 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:44:02.546 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-22T05:44:02.546 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-22T05:44:02.546 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:44:02.546 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-22T05:44:02.546 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-22T05:44:02.546 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:44:02.547 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-22T05:44:02.547 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-22T05:44:02.547 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:44:02.547 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-22T05:44:02.547 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-22T05:44:02.547 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-22T05:44:02.547 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-22T05:44:02.547 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-22T05:44:02.547 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:44:02.547 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:44:02.547 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-22T05:44:02.547 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-22T05:44:02.547 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-22T05:44:02.547 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-22T05:44:02.547 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-22T05:44:02.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-22T05:44:02.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-22T05:44:02.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-22T05:44:02.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-22T05:44:02.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-22T05:44:02.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-22T05:44:02.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-22T05:44:02.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-22T05:44:02.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-22T05:44:02.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:44:02.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:44:02.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-22T05:44:02.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-22T05:44:02.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-22T05:44:02.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-22T05:44:02.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-22T05:44:02.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-22T05:44:02.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-22T05:44:02.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-22T05:44:02.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-22T05:44:02.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-22T05:44:02.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-22T05:44:02.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:44:02.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:44:02.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:44:02.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:44:02.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:44:02.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:44:02.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:44:02.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:44:02.550 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:44:02.550 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:44:02.550 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:44:02.550 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:44:02.550 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-22T05:44:02.550 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-22T05:44:02.550 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 ' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-22T05:44:02.550 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:44:02.550 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:44:02.550 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:44:02.550 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:02 smithi186 ceph-mon[67903]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:44:02.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:44:02.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:44:02.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:44:02.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:44:02.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:44:02.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:44:02.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:44:02.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:44:02.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:44:02.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:44:02.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:44:02.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-22T05:44:02.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-22T05:44:02.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-03-22T05:44:02.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:44:02.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:44:02.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:44:02.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:44:02.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:44:02.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:44:02.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:44:02.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-22T05:44:02.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-22T05:44:02.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:44:02.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-22T05:44:02.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-22T05:44:02.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:44:02.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-22T05:44:02.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-22T05:44:02.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:44:02.705 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:44:02.705 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:44:02.705 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:44:02.705 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:44:02.705 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:44:02.705 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:44:02.705 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:44:02.705 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:44:02.705 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:44:02.705 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:44:02.705 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:44:02.705 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-22T05:44:02.705 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-22T05:44:02.705 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-03-22T05:44:02.706 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:44:02.706 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:44:02.706 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:44:02.706 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:44:02.706 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:44:02.706 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:44:02.706 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:44:02.706 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-22T05:44:02.706 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-22T05:44:02.706 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:44:02.706 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-22T05:44:02.706 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-22T05:44:02.706 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:44:02.707 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-22T05:44:02.707 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-22T05:44:02.707 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:44:02.707 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-22T05:44:02.707 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-22T05:44:02.707 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:44:02.707 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-22T05:44:02.707 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-22T05:44:02.707 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:44:02.707 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-22T05:44:02.708 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-22T05:44:02.708 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-22T05:44:02.708 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-22T05:44:02.708 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-22T05:44:02.708 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:44:02.708 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:44:02.708 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-22T05:44:02.708 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-22T05:44:02.708 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-22T05:44:02.708 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-22T05:44:02.708 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-22T05:44:02.708 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-22T05:44:02.708 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-22T05:44:02.708 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-22T05:44:02.708 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-22T05:44:02.709 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-22T05:44:02.709 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-22T05:44:02.709 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-22T05:44:02.709 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-22T05:44:02.709 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:44:02.709 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-22T05:44:02.709 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-22T05:44:02.709 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:44:02.709 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-22T05:44:02.710 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-22T05:44:02.710 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-22T05:44:02.710 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-22T05:44:02.710 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-22T05:44:02.710 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:44:02.710 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:44:02.710 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-22T05:44:02.710 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-22T05:44:02.710 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-22T05:44:02.710 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-22T05:44:02.710 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-22T05:44:02.710 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-22T05:44:02.710 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-22T05:44:02.711 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-22T05:44:02.711 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-22T05:44:02.711 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-22T05:44:02.711 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-22T05:44:02.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-22T05:44:02.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-22T05:44:02.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-22T05:44:02.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:44:02.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:44:02.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-22T05:44:02.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-22T05:44:02.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-22T05:44:02.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-22T05:44:02.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-22T05:44:02.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-22T05:44:02.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-22T05:44:02.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-22T05:44:02.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-22T05:44:02.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-22T05:44:02.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-22T05:44:02.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:44:02.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:44:02.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:44:02.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:44:02.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:44:02.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:44:02.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:44:02.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:44:02.714 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-22T05:44:02.714 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-22T05:44:02.715 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-22T05:44:02.715 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:44:02.715 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:44:02.715 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-22T05:44:02.715 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-22T05:44:02.715 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-22T05:44:02.715 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-22T05:44:02.715 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-22T05:44:02.715 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-22T05:44:02.715 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-22T05:44:02.715 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-22T05:44:02.715 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-22T05:44:02.715 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-22T05:44:02.715 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-22T05:44:02.715 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:44:02.716 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:44:02.716 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:44:02.716 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:44:02.716 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:44:02.716 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:44:02.716 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:44:02.716 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:44:02.716 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:44:02.716 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:44:02.716 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:44:02.716 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:44:02.716 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-22T05:44:02.716 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-22T05:44:02.716 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-22T05:44:02.716 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:44:02.716 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:44:02.717 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:44:02.717 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:44:02.717 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:44:02.717 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:44:02.717 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:44:02.717 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:44:02.717 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-22T05:44:02.717 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-22T05:44:02.717 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-22T05:44:02.717 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:44:02.717 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:44:02.717 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:44:02.718 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:02 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:44:03.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:03 smithi186 ceph-mon[67903]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:44:03.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:44:03.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:44:03.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:44:03.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:44:03.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:44:03.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:44:03.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:44:03.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:44:03.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:44:03.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:44:03.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:44:03.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:44:03.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:44:03.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:44:03.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:44:03.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:44:03.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:44:03.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:44:03.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:44:03.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:44:03.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:44:03.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:44:03.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:44:03.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:44:03.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:44:03.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:44:03.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:44:03.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:44:03.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: return future.result(timeout) 2024-03-22T05:44:03.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:44:03.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.__get_result() 2024-03-22T05:44:03.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:44:03.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: raise self._exception 2024-03-22T05:44:03.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:44:03.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:44:03.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:44:03.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:44:03.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:44:03.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:44:03.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:44:03.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:44:03.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:44:03.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:44:03.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:44:03.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:44:03.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:44:03.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:44:03.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:44:03.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:44:03.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:44:03.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:44:03.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:44:03.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:44:03.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:44:03.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:44:03.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:44:03.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:44:03.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:44:03.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:03 smithi186 ceph-mon[67903]: Upgrade: Setting container_image for all crash 2024-03-22T05:44:03.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:03 smithi186 ceph-mon[67903]: Upgrade: Setting container_image for all mds 2024-03-22T05:44:03.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:03 smithi186 ceph-mon[67903]: Upgrade: Setting container_image for all rbd-mirror 2024-03-22T05:44:03.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:03 smithi186 ceph-mon[67903]: Upgrade: Setting container_image for all cephfs-mirror 2024-03-22T05:44:03.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:03 smithi186 ceph-mon[67903]: Upgrade: Setting container_image for all ceph-exporter 2024-03-22T05:44:03.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:03 smithi186 ceph-mon[67903]: Upgrade: Setting container_image for all nfs 2024-03-22T05:44:03.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:03 smithi186 ceph-mon[67903]: Upgrade: Setting container_image for all nvmeof 2024-03-22T05:44:03.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:03 smithi186 ceph-mon[67903]: Upgrade: Setting container_image for all node-exporter 2024-03-22T05:44:03.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:03 smithi186 ceph-mon[67903]: Upgrade: Setting container_image for all prometheus 2024-03-22T05:44:03.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:03 smithi186 ceph-mon[67903]: Upgrade: Setting container_image for all alertmanager 2024-03-22T05:44:03.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:03 smithi186 ceph-mon[67903]: Upgrade: Setting container_image for all grafana 2024-03-22T05:44:03.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:03 smithi186 ceph-mon[67903]: Upgrade: Setting container_image for all loki 2024-03-22T05:44:03.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:03 smithi186 ceph-mon[67903]: Upgrade: Setting container_image for all promtail 2024-03-22T05:44:03.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:03 smithi186 ceph-mon[67903]: Upgrade: Finalizing container_image settings 2024-03-22T05:44:03.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:03 smithi186 ceph-mon[67903]: Upgrade: Complete! 2024-03-22T05:44:03.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:03 smithi186 ceph-mon[67903]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-22T05:44:03.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:03 smithi186 ceph-mon[67903]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 664 B/s rd, 0 op/s 2024-03-22T05:44:03.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:03 smithi186 ceph-mon[67903]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:44:03.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:03 smithi186 ceph-mon[67903]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:44:03.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:03 smithi186 ceph-mon[67903]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:44:03.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:03 smithi186 ceph-mon[67903]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:44:03.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:03 smithi186 ceph-mon[67903]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:44:03.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:03 smithi186 ceph-mon[67903]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:44:03.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:03 smithi005 ceph-mon[24793]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:44:03.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:44:03.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:44:03.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:44:03.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:44:03.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:44:03.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:44:03.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:44:03.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:44:03.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:44:03.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:44:03.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:44:03.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:44:03.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:44:03.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:44:03.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:44:03.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:44:03.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:44:03.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:44:03.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:44:03.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:44:03.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:44:03.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:44:03.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:44:03.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:44:03.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:44:03.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:44:03.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:44:03.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:44:03.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: return future.result(timeout) 2024-03-22T05:44:03.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:44:03.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.__get_result() 2024-03-22T05:44:03.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:44:03.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: raise self._exception 2024-03-22T05:44:03.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:44:03.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:44:03.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:44:03.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:44:03.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:44:03.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:44:03.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:44:03.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:44:03.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:44:03.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:44:03.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:44:03.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:44:03.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:44:03.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:44:03.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:44:03.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:44:03.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:44:03.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:44:03.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:44:03.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:44:03.706 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:03 smithi005 ceph-mon[29523]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:44:03.706 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:44:03.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:44:03.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:44:03.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:44:03.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:44:03.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:44:03.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:44:03.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:44:03.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:44:03.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:44:03.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:44:03.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:44:03.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:44:03.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:44:03.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:44:03.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:44:03.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:44:03.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:44:03.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:44:03.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:44:03.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:44:03.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:44:03.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:44:03.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:44:03.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:44:03.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:44:03.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:44:03.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:44:03.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: return future.result(timeout) 2024-03-22T05:44:03.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:44:03.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.__get_result() 2024-03-22T05:44:03.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:44:03.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: raise self._exception 2024-03-22T05:44:03.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:44:03.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:44:03.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:44:03.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:44:03.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:44:03.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:44:03.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:44:03.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:44:03.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:44:03.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:44:03.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:44:03.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:44:03.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:44:03.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:44:03.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:44:03.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:44:03.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:44:03.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:44:03.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:44:03.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:44:03.710 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:44:03.710 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:44:03.710 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:44:03.711 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:44:03.711 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:44:03.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:03 smithi005 ceph-mon[24793]: Upgrade: Setting container_image for all crash 2024-03-22T05:44:03.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:03 smithi005 ceph-mon[24793]: Upgrade: Setting container_image for all mds 2024-03-22T05:44:03.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:03 smithi005 ceph-mon[24793]: Upgrade: Setting container_image for all rbd-mirror 2024-03-22T05:44:03.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:03 smithi005 ceph-mon[24793]: Upgrade: Setting container_image for all cephfs-mirror 2024-03-22T05:44:03.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:03 smithi005 ceph-mon[24793]: Upgrade: Setting container_image for all ceph-exporter 2024-03-22T05:44:03.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:03 smithi005 ceph-mon[24793]: Upgrade: Setting container_image for all nfs 2024-03-22T05:44:03.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:03 smithi005 ceph-mon[24793]: Upgrade: Setting container_image for all nvmeof 2024-03-22T05:44:03.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:03 smithi005 ceph-mon[24793]: Upgrade: Setting container_image for all node-exporter 2024-03-22T05:44:03.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:03 smithi005 ceph-mon[24793]: Upgrade: Setting container_image for all prometheus 2024-03-22T05:44:03.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:03 smithi005 ceph-mon[24793]: Upgrade: Setting container_image for all alertmanager 2024-03-22T05:44:03.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:03 smithi005 ceph-mon[24793]: Upgrade: Setting container_image for all grafana 2024-03-22T05:44:03.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:03 smithi005 ceph-mon[24793]: Upgrade: Setting container_image for all loki 2024-03-22T05:44:03.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:03 smithi005 ceph-mon[24793]: Upgrade: Setting container_image for all promtail 2024-03-22T05:44:03.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:03 smithi005 ceph-mon[24793]: Upgrade: Finalizing container_image settings 2024-03-22T05:44:03.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:03 smithi005 ceph-mon[24793]: Upgrade: Complete! 2024-03-22T05:44:03.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:03 smithi005 ceph-mon[24793]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-22T05:44:03.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:03 smithi005 ceph-mon[24793]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 664 B/s rd, 0 op/s 2024-03-22T05:44:03.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:03 smithi005 ceph-mon[24793]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:44:03.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:03 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:44:03.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:03 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:44:03.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:03 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:44:03.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:03 smithi005 ceph-mon[24793]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:44:03.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:03 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:44:03.712 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:44:03.712 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:44:03.713 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:44:03.713 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:44:03.713 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:44:03.713 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:03 smithi005 ceph-mon[29523]: Upgrade: Setting container_image for all crash 2024-03-22T05:44:03.713 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:03 smithi005 ceph-mon[29523]: Upgrade: Setting container_image for all mds 2024-03-22T05:44:03.713 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:03 smithi005 ceph-mon[29523]: Upgrade: Setting container_image for all rbd-mirror 2024-03-22T05:44:03.713 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:03 smithi005 ceph-mon[29523]: Upgrade: Setting container_image for all cephfs-mirror 2024-03-22T05:44:03.713 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:03 smithi005 ceph-mon[29523]: Upgrade: Setting container_image for all ceph-exporter 2024-03-22T05:44:03.713 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:03 smithi005 ceph-mon[29523]: Upgrade: Setting container_image for all nfs 2024-03-22T05:44:03.713 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:03 smithi005 ceph-mon[29523]: Upgrade: Setting container_image for all nvmeof 2024-03-22T05:44:03.713 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:03 smithi005 ceph-mon[29523]: Upgrade: Setting container_image for all node-exporter 2024-03-22T05:44:03.713 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:03 smithi005 ceph-mon[29523]: Upgrade: Setting container_image for all prometheus 2024-03-22T05:44:03.713 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:03 smithi005 ceph-mon[29523]: Upgrade: Setting container_image for all alertmanager 2024-03-22T05:44:03.713 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:03 smithi005 ceph-mon[29523]: Upgrade: Setting container_image for all grafana 2024-03-22T05:44:03.713 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:03 smithi005 ceph-mon[29523]: Upgrade: Setting container_image for all loki 2024-03-22T05:44:03.714 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:03 smithi005 ceph-mon[29523]: Upgrade: Setting container_image for all promtail 2024-03-22T05:44:03.714 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:03 smithi005 ceph-mon[29523]: Upgrade: Finalizing container_image settings 2024-03-22T05:44:03.714 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:03 smithi005 ceph-mon[29523]: Upgrade: Complete! 2024-03-22T05:44:03.714 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:03 smithi005 ceph-mon[29523]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-22T05:44:03.714 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:03 smithi005 ceph-mon[29523]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 664 B/s rd, 0 op/s 2024-03-22T05:44:03.714 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:03 smithi005 ceph-mon[29523]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:44:03.714 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:03 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:44:03.714 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:03 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:44:03.714 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:03 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:44:03.714 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:03 smithi005 ceph-mon[29523]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:44:03.714 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:03 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:44:04.430 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:44:04.152+0000 7f487c563700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:44:04.430 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:44:04.430 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:44:04.430 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:44:04.430 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:44:04.431 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:44:04.431 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:44:04.431 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:44:04.431 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:44:04.431 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:44:04.431 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:44:04.431 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:44:04.431 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:44:04.431 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:44:04.431 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:44:04.431 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:44:04.431 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:44:04.431 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:44:04.431 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:44:04.432 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:44:04.432 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:44:04.432 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:44:04.432 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:44:04.432 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:44:04.432 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:44:04.432 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:44:04.433 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:44:04.433 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:44:04.433 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:44:04.433 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return future.result(timeout) 2024-03-22T05:44:04.433 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:44:04.433 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.__get_result() 2024-03-22T05:44:04.433 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:44:04.433 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: raise self._exception 2024-03-22T05:44:04.433 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:44:04.433 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: config_blobs=daemon_spec.final_config, 2024-03-22T05:44:04.433 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:44:04.433 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:44:04.433 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:44:04.433 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:44:04.433 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:44:04.433 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:44:04.434 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:44:04.434 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:44:04.434 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:44:04.434 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:44:04.434 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:44:04.434 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:44:04.434 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:44:04.434 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:44:04.434 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:44:04.434 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:44:04.434 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:44:04.434 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:44:04.434 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:44:04.434 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:44:04.435 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:44:04.435 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:44:04.435 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:44:04.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:04 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:44:04.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:04 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:44:04.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:04 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:44:04.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:04 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:44:04.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:04 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:44:04.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:04 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:44:04.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:04 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:44:04.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:04 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:44:04.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:04 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:44:04.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:04 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:44:04.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:04 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:44:04.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:04 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:44:04.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:04 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:44:04.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:04 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:44:04.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:04 smithi186 ceph-mon[67903]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:44:04.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:04 smithi186 ceph-mon[67903]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:44:04.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:04 smithi186 ceph-mon[67903]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:44:04.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:04 smithi186 ceph-mon[67903]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:44:04.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:04 smithi186 ceph-mon[67903]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:44:04.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:04 smithi186 ceph-mon[67903]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:44:04.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:04 smithi186 ceph-mon[67903]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:44:05.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:05 smithi005 ceph-mon[24793]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:44:05.702 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:44:05.702 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:44:05.702 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:44:05.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:44:05.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:44:05.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:44:05.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:44:05.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:44:05.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:44:05.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:44:05.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:44:05.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:44:05.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:44:05.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:44:05.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:44:05.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:44:05.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:44:05.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:44:05.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:44:05.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:44:05.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:44:05.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:44:05.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:44:05.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:44:05.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:44:05.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:44:05.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:44:05.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:44:05.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: return future.result(timeout) 2024-03-22T05:44:05.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:44:05.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.__get_result() 2024-03-22T05:44:05.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:44:05.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: raise self._exception 2024-03-22T05:44:05.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:44:05.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:44:05.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:44:05.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:44:05.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:44:05.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:44:05.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:44:05.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:44:05.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:44:05.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:44:05.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:44:05.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:44:05.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:44:05.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:44:05.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:44:05.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:44:05.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:44:05.707 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:44:05.707 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:44:05.707 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:44:05.707 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:44:05.707 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:44:05.707 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:44:05.707 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:44:05.707 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:44:05.707 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:05 smithi005 ceph-mon[24793]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-22T05:44:05.708 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:05 smithi005 ceph-mon[24793]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-03-22T05:44:05.708 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:05 smithi005 ceph-mon[24793]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:44:05.708 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:05 smithi005 ceph-mon[24793]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:44:05.708 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:05 smithi005 ceph-mon[29523]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:44:05.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:44:05.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:44:05.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:44:05.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:44:05.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:44:05.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:44:05.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:44:05.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:44:05.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:44:05.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:44:05.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:44:05.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:44:05.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:44:05.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:44:05.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:44:05.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:44:05.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:44:05.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:44:05.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:44:05.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:44:05.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:44:05.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:44:05.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:44:05.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:44:05.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:44:05.711 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:44:05.711 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:44:05.711 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:44:05.711 INFO:journalctl@ceph.mon.c.smithi005.stdout: return future.result(timeout) 2024-03-22T05:44:05.711 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:44:05.711 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.__get_result() 2024-03-22T05:44:05.711 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:44:05.711 INFO:journalctl@ceph.mon.c.smithi005.stdout: raise self._exception 2024-03-22T05:44:05.711 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:44:05.711 INFO:journalctl@ceph.mon.c.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:44:05.711 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:44:05.712 INFO:journalctl@ceph.mon.c.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:44:05.712 INFO:journalctl@ceph.mon.c.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:44:05.712 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:44:05.712 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:44:05.712 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:44:05.712 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:44:05.712 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:44:05.712 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:44:05.712 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:44:05.712 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:44:05.712 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:44:05.712 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:44:05.713 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:44:05.713 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:44:05.713 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:44:05.713 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:44:05.713 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:44:05.713 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:44:05.713 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:44:05.713 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:44:05.713 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:44:05.713 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:44:05.713 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:05 smithi005 ceph-mon[29523]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-22T05:44:05.713 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:05 smithi005 ceph-mon[29523]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-03-22T05:44:05.713 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:05 smithi005 ceph-mon[29523]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:44:05.713 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:05 smithi005 ceph-mon[29523]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:44:05.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:05 smithi186 ceph-mon[67903]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:44:05.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:44:05.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:44:05.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:44:05.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:44:05.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:44:05.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:44:05.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:44:05.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:44:05.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:44:05.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:44:05.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:44:05.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:44:05.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:44:05.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:44:05.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:44:05.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:44:05.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:44:05.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:44:05.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:44:05.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:44:05.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:44:05.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:44:05.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:44:05.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:44:05.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:44:05.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:44:05.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:44:05.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:44:05.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: return future.result(timeout) 2024-03-22T05:44:05.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:44:05.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.__get_result() 2024-03-22T05:44:05.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:44:05.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: raise self._exception 2024-03-22T05:44:05.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:44:05.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:44:05.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:44:05.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:44:05.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:44:05.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:44:05.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:44:05.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:44:05.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:44:05.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:44:05.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:44:05.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:44:05.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:44:05.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:44:05.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:44:05.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:44:05.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:44:05.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:44:05.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:44:05.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:44:05.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:44:05.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:44:05.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:44:05.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:44:05.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:44:05.798 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:05 smithi186 ceph-mon[67903]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-22T05:44:05.798 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:05 smithi186 ceph-mon[67903]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-03-22T05:44:05.798 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:05 smithi186 ceph-mon[67903]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:44:05.798 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:05 smithi186 ceph-mon[67903]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:44:06.452 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:44:06.169+0000 7f487c563700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:44:06.452 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:44:06.453 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:44:06.453 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:44:06.453 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:44:06.453 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:44:06.453 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:44:06.453 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:44:06.453 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:44:06.453 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:44:06.453 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:44:06.453 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:44:06.453 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:44:06.453 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:44:06.453 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:44:06.453 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:44:06.454 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:44:06.454 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:44:06.454 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:44:06.454 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:44:06.454 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:44:06.454 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:44:06.454 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:44:06.455 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:44:06.455 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:44:06.455 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:44:06.455 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:44:06.455 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:44:06.455 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:44:06.455 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return future.result(timeout) 2024-03-22T05:44:06.455 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:44:06.455 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.__get_result() 2024-03-22T05:44:06.455 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:44:06.455 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: raise self._exception 2024-03-22T05:44:06.455 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:44:06.455 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: config_blobs=daemon_spec.final_config, 2024-03-22T05:44:06.455 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:44:06.456 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:44:06.456 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:44:06.456 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:44:06.456 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:44:06.456 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:44:06.456 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:44:06.456 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:44:06.456 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:44:06.456 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:44:06.456 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:44:06.456 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:44:06.456 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:44:06.456 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:44:06.456 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:44:06.456 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:44:06.457 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:44:06.457 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:44:06.457 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:44:06.457 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:44:06.457 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:44:06.457 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:44:06.457 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:44:07.433 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:44:06] "GET /metrics HTTP/1.1" 200 37931 "" "Prometheus/2.43.0" 2024-03-22T05:44:07.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:07 smithi005 ceph-mon[29523]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 859 B/s rd, 0 op/s 2024-03-22T05:44:07.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:07 smithi005 ceph-mon[29523]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:44:07.702 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:44:07.702 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:44:07.703 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:44:07.703 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:44:07.703 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:44:07.703 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:44:07.703 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:44:07.703 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:44:07.703 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:44:07.703 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:44:07.703 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:44:07.703 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:44:07.703 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:44:07.703 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:44:07.703 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:44:07.703 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:44:07.704 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:44:07.704 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:44:07.704 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:44:07.704 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:44:07.704 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:44:07.704 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:44:07.704 INFO:journalctl@ceph.mon.c.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:44:07.704 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:44:07.704 INFO:journalctl@ceph.mon.c.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:44:07.704 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:44:07.704 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:44:07.704 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:44:07.704 INFO:journalctl@ceph.mon.c.smithi005.stdout: return future.result(timeout) 2024-03-22T05:44:07.704 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:44:07.704 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.__get_result() 2024-03-22T05:44:07.704 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:44:07.705 INFO:journalctl@ceph.mon.c.smithi005.stdout: raise self._exception 2024-03-22T05:44:07.705 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:44:07.705 INFO:journalctl@ceph.mon.c.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:44:07.705 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:44:07.705 INFO:journalctl@ceph.mon.c.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:44:07.705 INFO:journalctl@ceph.mon.c.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:44:07.705 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:44:07.705 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:44:07.705 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:44:07.705 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:44:07.705 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:44:07.705 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:44:07.705 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:44:07.705 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:44:07.705 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:44:07.706 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:44:07.706 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:44:07.706 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:44:07.706 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:44:07.706 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:44:07.706 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:44:07.706 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:44:07.706 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:44:07.706 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:44:07.706 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:44:07.706 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:44:07.707 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:07 smithi005 ceph-mon[24793]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 859 B/s rd, 0 op/s 2024-03-22T05:44:07.707 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:07 smithi005 ceph-mon[24793]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:44:07.707 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:44:07.707 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:44:07.707 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:44:07.707 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:44:07.707 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:44:07.707 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:44:07.707 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:44:07.707 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:44:07.707 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:44:07.707 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:44:07.708 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:44:07.708 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:44:07.708 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:44:07.708 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:44:07.708 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:44:07.708 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:44:07.708 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:44:07.708 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:44:07.708 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:44:07.708 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:44:07.708 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:44:07.708 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:44:07.708 INFO:journalctl@ceph.mon.a.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:44:07.708 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:44:07.709 INFO:journalctl@ceph.mon.a.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:44:07.709 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:44:07.709 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:44:07.709 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:44:07.709 INFO:journalctl@ceph.mon.a.smithi005.stdout: return future.result(timeout) 2024-03-22T05:44:07.709 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:44:07.709 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.__get_result() 2024-03-22T05:44:07.709 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:44:07.709 INFO:journalctl@ceph.mon.a.smithi005.stdout: raise self._exception 2024-03-22T05:44:07.709 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:44:07.709 INFO:journalctl@ceph.mon.a.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:44:07.709 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:44:07.709 INFO:journalctl@ceph.mon.a.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:44:07.709 INFO:journalctl@ceph.mon.a.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:44:07.710 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:44:07.710 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:44:07.710 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:44:07.710 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:44:07.710 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:44:07.710 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:44:07.710 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:44:07.710 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:44:07.710 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:44:07.710 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:44:07.710 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:44:07.710 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:44:07.710 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:44:07.710 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:44:07.711 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:44:07.711 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:44:07.711 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:44:07.711 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:44:07.711 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:44:07.711 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:44:07.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:07 smithi186 ceph-mon[67903]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 859 B/s rd, 0 op/s 2024-03-22T05:44:07.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:07 smithi186 ceph-mon[67903]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:44:07.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:44:07.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:44:07.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:44:07.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:44:07.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:44:07.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:44:07.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:44:07.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:44:07.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:44:07.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:44:07.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:44:07.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:44:07.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:44:07.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:44:07.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:44:07.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:44:07.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:44:07.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:44:07.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:44:07.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:44:07.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:44:07.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:44:07.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:44:07.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:44:07.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:44:07.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:44:07.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:44:07.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:44:07.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: return future.result(timeout) 2024-03-22T05:44:07.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:44:07.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.__get_result() 2024-03-22T05:44:07.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:44:07.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: raise self._exception 2024-03-22T05:44:07.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:44:07.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:44:07.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:44:07.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:44:07.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:44:07.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:44:07.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:44:07.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:44:07.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:44:07.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:44:07.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:44:07.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:44:07.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:44:07.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:44:07.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:44:07.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:44:07.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:44:07.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:44:07.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:44:07.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:44:07.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:44:07.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:44:07.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:44:07.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:44:07.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:44:08.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:08 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:44:08.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:08 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:44:08.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:08 smithi186 ceph-mon[67903]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:44:09.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:09 smithi005 ceph-mon[24793]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 643 B/s rd, 0 op/s 2024-03-22T05:44:09.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:09 smithi005 ceph-mon[29523]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 643 B/s rd, 0 op/s 2024-03-22T05:44:09.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:09 smithi186 ceph-mon[67903]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 643 B/s rd, 0 op/s 2024-03-22T05:44:11.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:11 smithi005 ceph-mon[24793]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-22T05:44:11.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:11 smithi005 ceph-mon[29523]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-22T05:44:11.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:11 smithi186 ceph-mon[67903]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-22T05:44:12.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:12 smithi005 ceph-mon[24793]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:44:12.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:12 smithi005 ceph-mon[29523]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:44:12.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:12 smithi186 ceph-mon[67903]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:44:13.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:13 smithi005 ceph-mon[24793]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-22T05:44:13.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:13 smithi005 ceph-mon[29523]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-22T05:44:13.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:13 smithi186 ceph-mon[67903]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-22T05:44:15.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:15 smithi005 ceph-mon[24793]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:44:15.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:15 smithi005 ceph-mon[29523]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:44:15.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:15 smithi186 ceph-mon[67903]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:44:16.114 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph orch ps' 2024-03-22T05:44:16.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:16 smithi005 ceph-mon[24793]: from='client.15636 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:44:16.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:16 smithi005 ceph-mon[29523]: from='client.15636 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:44:16.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:16 smithi186 ceph-mon[67903]: from='client.15636 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:44:17.110 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:16 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:44:16] "GET /metrics HTTP/1.1" 200 38000 "" "Prometheus/2.43.0" 2024-03-22T05:44:17.672 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:17 smithi005 ceph-mon[24793]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:44:17.673 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:17 smithi005 ceph-mon[29523]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:44:17.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:17 smithi186 ceph-mon[67903]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:44:18.148 INFO:teuthology.orchestra.run.smithi005.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T05:44:18.148 INFO:teuthology.orchestra.run.smithi005.stdout:alertmanager.a smithi005 *:9093,9094 running (112s) 107s ago 23m 17.1M - 0.25.0 c8568f914cd2 ee99dfc435e8 2024-03-22T05:44:18.149 INFO:teuthology.orchestra.run.smithi005.stdout:grafana.a smithi186 *:3000 running (95s) 22s ago 23m 75.7M - 9.4.12 2bacad6d85d8 16fd57dbbd71 2024-03-22T05:44:18.149 INFO:teuthology.orchestra.run.smithi005.stdout:iscsi.foo.smithi005.mcdwbp smithi005 unknown 107s ago 22m 58.9M - 3.5 e1d6a67b021e 5215e03bf81b 2024-03-22T05:44:18.149 INFO:teuthology.orchestra.run.smithi005.stdout:mgr.x smithi186 *:8443,9283,8765 running (2m) 22s ago 28m 450M - 19.0.0-2299-gea01217f aa461e7c5e7f c556421a73f3 2024-03-22T05:44:18.149 INFO:teuthology.orchestra.run.smithi005.stdout:mgr.y smithi005 *:8443,9283,8765 running (12m) 107s ago 29m 503M - 19.0.0-2299-gea01217f aa461e7c5e7f 62a813eebc0d 2024-03-22T05:44:18.149 INFO:teuthology.orchestra.run.smithi005.stdout:mon.a smithi005 running (29m) 107s ago 29m 78.5M 2048M 17.2.0 e1d6a67b021e 38e0bba4bb03 2024-03-22T05:44:18.149 INFO:teuthology.orchestra.run.smithi005.stdout:mon.b smithi186 running (26s) 22s ago 28m 20.9M 2048M 19.0.0-2299-gea01217f aa461e7c5e7f c1798feac15b 2024-03-22T05:44:18.149 INFO:teuthology.orchestra.run.smithi005.stdout:mon.c smithi005 running (28m) 107s ago 28m 58.8M 2048M 17.2.0 e1d6a67b021e 00ec24d38612 2024-03-22T05:44:18.149 INFO:teuthology.orchestra.run.smithi005.stdout:node-exporter.a smithi005 *:9100 running (2m) 107s ago 24m 8786k - 1.5.0 0da6a335fe13 f635cd72b9d3 2024-03-22T05:44:18.150 INFO:teuthology.orchestra.run.smithi005.stdout:node-exporter.b smithi186 *:9100 running (2m) 22s ago 24m 14.5M - 1.5.0 0da6a335fe13 d7e0b3d20191 2024-03-22T05:44:18.150 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0 smithi005 running (27m) 107s ago 27m 53.5M 2993M 17.2.0 e1d6a67b021e ef5c7053dd37 2024-03-22T05:44:18.150 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1 smithi005 running (27m) 107s ago 27m 56.5M 2993M 17.2.0 e1d6a67b021e a1e7ee1017b6 2024-03-22T05:44:18.150 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2 smithi005 running (27m) 107s ago 27m 48.5M 2993M 17.2.0 e1d6a67b021e 462ea63ad0e5 2024-03-22T05:44:18.150 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3 smithi005 running (26m) 107s ago 26m 56.5M 2993M 17.2.0 e1d6a67b021e b25432a7239b 2024-03-22T05:44:18.150 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4 smithi186 running (26m) 22s ago 26m 56.4M 3248M 17.2.0 e1d6a67b021e cd963671f340 2024-03-22T05:44:18.150 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5 smithi186 running (26m) 22s ago 26m 51.6M 3248M 17.2.0 e1d6a67b021e f98c87f1a2cd 2024-03-22T05:44:18.150 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6 smithi186 running (25m) 22s ago 25m 52.0M 3248M 17.2.0 e1d6a67b021e 704ac49c68c4 2024-03-22T05:44:18.150 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7 smithi186 running (25m) 22s ago 25m 54.4M 3248M 17.2.0 e1d6a67b021e f415b90edf63 2024-03-22T05:44:18.150 INFO:teuthology.orchestra.run.smithi005.stdout:prometheus.a smithi186 *:9095 running (2m) 22s ago 24m 66.2M - 2.43.0 a07b618ecd1d a5a043588d6d 2024-03-22T05:44:18.151 INFO:teuthology.orchestra.run.smithi005.stdout:rgw.foo.smithi005.exdhqu smithi005 *:8000 running (23m) 107s ago 23m 78.8M - 17.2.0 e1d6a67b021e 6bb27fe11553 2024-03-22T05:44:18.151 INFO:teuthology.orchestra.run.smithi005.stdout:rgw.foo.smithi186.mcgodk smithi186 *:8000 running (22m) 22s ago 22m 77.9M - 17.2.0 e1d6a67b021e 19b5c20a34f7 2024-03-22T05:44:18.852 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph versions | jq -e '"'"'.mon | length == 2'"'"'' 2024-03-22T05:44:19.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:19 smithi005 ceph-mon[29523]: from='client.25543 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:44:19.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:19 smithi005 ceph-mon[29523]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:44:19.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:19 smithi005 ceph-mon[24793]: from='client.25543 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:44:19.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:19 smithi005 ceph-mon[24793]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:44:19.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:19 smithi186 ceph-mon[67903]: from='client.25543 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:44:19.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:19 smithi186 ceph-mon[67903]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:44:20.999 INFO:teuthology.orchestra.run.smithi005.stdout:true 2024-03-22T05:44:21.574 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:21 smithi005 ceph-mon[24793]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:44:21.575 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:21 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/2287011007' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:44:21.575 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:21 smithi005 ceph-mon[29523]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:44:21.575 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:21 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/2287011007' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:44:21.599 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph orch upgrade status' 2024-03-22T05:44:21.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:21 smithi186 ceph-mon[67903]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:44:21.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:21 smithi186 ceph-mon[67903]: from='client.? 172.21.15.5:0/2287011007' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:44:22.581 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:22 smithi005 ceph-mon[24793]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:44:22.581 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:22 smithi005 ceph-mon[29523]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:44:22.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:22 smithi186 ceph-mon[67903]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:44:23.652 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-03-22T05:44:23.652 INFO:teuthology.orchestra.run.smithi005.stdout: "target_image": null, 2024-03-22T05:44:23.652 INFO:teuthology.orchestra.run.smithi005.stdout: "in_progress": false, 2024-03-22T05:44:23.652 INFO:teuthology.orchestra.run.smithi005.stdout: "which": "", 2024-03-22T05:44:23.652 INFO:teuthology.orchestra.run.smithi005.stdout: "services_complete": [], 2024-03-22T05:44:23.652 INFO:teuthology.orchestra.run.smithi005.stdout: "progress": null, 2024-03-22T05:44:23.653 INFO:teuthology.orchestra.run.smithi005.stdout: "message": "", 2024-03-22T05:44:23.653 INFO:teuthology.orchestra.run.smithi005.stdout: "is_paused": false 2024-03-22T05:44:23.653 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-03-22T05:44:23.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:23 smithi186 ceph-mon[67903]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:44:23.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:23 smithi186 ceph-mon[67903]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:44:23.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:23 smithi005 ceph-mon[24793]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:44:23.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:23 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:44:23.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:23 smithi005 ceph-mon[29523]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:44:23.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:23 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:44:24.331 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph health detail' 2024-03-22T05:44:24.644 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:24 smithi005 ceph-mon[29523]: from='client.15654 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:44:24.645 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:24 smithi005 ceph-mon[24793]: from='client.15654 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:44:24.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:24 smithi186 ceph-mon[67903]: from='client.15654 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:44:25.736 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:25 smithi005 ceph-mon[29523]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:44:25.736 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:25 smithi005 ceph-mon[24793]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:44:25.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:25 smithi186 ceph-mon[67903]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:44:26.358 INFO:teuthology.orchestra.run.smithi005.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-22T05:44:26.358 INFO:teuthology.orchestra.run.smithi005.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-22T05:44:26.358 INFO:teuthology.orchestra.run.smithi005.stdout: daemon iscsi.foo.smithi005.mcdwbp on smithi005 is in unknown state 2024-03-22T05:44:26.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:26 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/4029594634' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T05:44:26.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:26 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/4029594634' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T05:44:26.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:26 smithi186 ceph-mon[67903]: from='client.? 172.21.15.5:0/4029594634' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T05:44:27.113 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:26 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:44:26] "GET /metrics HTTP/1.1" 200 37995 "" "Prometheus/2.43.0" 2024-03-22T05:44:27.139 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mon --hosts $(ceph orch ps | grep mgr.y | awk '"'"'{print $2}'"'"')' 2024-03-22T05:44:27.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:27 smithi005 ceph-mon[24793]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:44:27.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:27 smithi005 ceph-mon[29523]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:44:27.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:27 smithi186 ceph-mon[67903]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:44:29.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:29 smithi186 ceph-mon[67903]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:44:29.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:29 smithi005 ceph-mon[24793]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:44:29.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:29 smithi005 ceph-mon[29523]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:44:30.746 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:30 smithi186 ceph-mon[67903]: from='client.15666 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:44:30.746 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:30 smithi186 ceph-mon[67903]: from='client.25573 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37", "daemon_types": "mon", "hosts": "smithi005", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:44:30.763 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:30 smithi005 ceph-mon[24793]: from='client.15666 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:44:30.763 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:30 smithi005 ceph-mon[24793]: from='client.25573 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37", "daemon_types": "mon", "hosts": "smithi005", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:44:30.764 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:30 smithi005 ceph-mon[29523]: from='client.15666 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:44:30.764 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:30 smithi005 ceph-mon[29523]: from='client.25573 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37", "daemon_types": "mon", "hosts": "smithi005", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:44:31.639 INFO:teuthology.orchestra.run.smithi005.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37 2024-03-22T05:44:31.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:31 smithi186 ceph-mon[67903]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:44:31.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:31 smithi005 ceph-mon[24793]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:44:31.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:31 smithi005 ceph-mon[29523]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:44:32.444 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-03-22T05:44:32.682 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:32 smithi005 ceph-mon[24793]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37 2024-03-22T05:44:32.683 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:32 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:44:32.683 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:32 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:44:32.683 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:32 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:44:32.683 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:32 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:44:32.683 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:32 smithi005 ceph-mon[24793]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 892 B/s rd, 0 op/s 2024-03-22T05:44:32.683 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:32 smithi005 ceph-mon[24793]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:44:32.683 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:32 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:44:32.683 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:32 smithi005 ceph-mon[24793]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:44:32.683 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:32 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:44:32.683 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:32 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:44:32.683 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:32 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:44:32.684 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:32 smithi005 ceph-mon[29523]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37 2024-03-22T05:44:32.684 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:32 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:44:32.684 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:32 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:44:32.684 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:32 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:44:32.684 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:32 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:44:32.684 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:32 smithi005 ceph-mon[29523]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 892 B/s rd, 0 op/s 2024-03-22T05:44:32.684 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:32 smithi005 ceph-mon[29523]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:44:32.684 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:32 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:44:32.684 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:32 smithi005 ceph-mon[29523]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:44:32.684 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:32 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:44:32.684 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:32 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:44:32.684 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:32 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:44:33.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:32 smithi186 ceph-mon[67903]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37 2024-03-22T05:44:33.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:32 smithi186 ceph-mon[67903]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:44:33.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:32 smithi186 ceph-mon[67903]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:44:33.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:32 smithi186 ceph-mon[67903]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:44:33.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:32 smithi186 ceph-mon[67903]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:44:33.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:32 smithi186 ceph-mon[67903]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 892 B/s rd, 0 op/s 2024-03-22T05:44:33.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:32 smithi186 ceph-mon[67903]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:44:33.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:32 smithi186 ceph-mon[67903]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:44:33.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:32 smithi186 ceph-mon[67903]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:44:33.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:32 smithi186 ceph-mon[67903]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:44:33.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:32 smithi186 ceph-mon[67903]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:44:33.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:32 smithi186 ceph-mon[67903]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:44:33.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:33 smithi005 ceph-mon[24793]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:44:33.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:33 smithi005 ceph-mon[24793]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:44:33.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:33 smithi005 ceph-mon[29523]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:44:33.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:33 smithi005 ceph-mon[29523]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:44:34.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:33 smithi186 ceph-mon[67903]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:44:34.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:33 smithi186 ceph-mon[67903]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:44:34.647 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:44:34.454+0000 7f487c563700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:44:34.648 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:44:34.648 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:44:34.648 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:44:34.648 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:44:34.648 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:44:34.648 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:44:34.648 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:44:34.648 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:44:34.648 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:44:34.648 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:44:34.648 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:44:34.649 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:44:34.649 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:44:34.649 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:44:34.649 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:44:34.649 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:44:34.649 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:44:34.649 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:44:34.649 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:44:34.650 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:44:34.650 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:44:34.650 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:44:34.650 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:44:34.650 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:44:34.650 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:44:34.650 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:44:34.650 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:44:34.650 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:44:34.650 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return future.result(timeout) 2024-03-22T05:44:34.650 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:44:34.650 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.__get_result() 2024-03-22T05:44:34.650 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:44:34.650 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: raise self._exception 2024-03-22T05:44:34.651 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:44:34.651 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: config_blobs=daemon_spec.final_config, 2024-03-22T05:44:34.651 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:44:34.651 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:44:34.651 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:44:34.651 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:44:34.651 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:44:34.651 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:44:34.651 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:44:34.651 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:44:34.651 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:44:34.651 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:44:34.651 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:44:34.651 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:44:34.652 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:44:34.652 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:44:34.652 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:44:34.652 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:44:34.652 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:44:34.652 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:44:34.652 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:44:34.652 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:44:34.652 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:44:34.652 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:44:34.652 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:44:34.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:34 smithi005 ceph-mon[24793]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 540 B/s rd, 0 op/s 2024-03-22T05:44:34.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:34 smithi005 ceph-mon[29523]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 540 B/s rd, 0 op/s 2024-03-22T05:44:35.046 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:34 smithi186 ceph-mon[67903]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 540 B/s rd, 0 op/s 2024-03-22T05:44:35.581 INFO:teuthology.orchestra.run.smithi005.stdout:true 2024-03-22T05:44:35.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:35 smithi005 ceph-mon[24793]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:44:35.952 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:44:35.952 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:44:35.952 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:44:35.952 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:44:35.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:44:35.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:44:35.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:44:35.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:44:35.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:44:35.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:44:35.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:44:35.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:44:35.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:44:35.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:44:35.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:44:35.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:44:35.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:44:35.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:44:35.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:44:35.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:44:35.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:44:35.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:44:35.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:44:35.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:44:35.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:44:35.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:44:35.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:44:35.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:44:35.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: return future.result(timeout) 2024-03-22T05:44:35.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:44:35.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.__get_result() 2024-03-22T05:44:35.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:44:35.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: raise self._exception 2024-03-22T05:44:35.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:44:35.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:44:35.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:44:35.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:44:35.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:44:35.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:44:35.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:44:35.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:44:35.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:44:35.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:44:35.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:44:35.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:44:35.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:44:35.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:44:35.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:44:35.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:44:35.956 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:44:35.956 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:44:35.956 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:44:35.956 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:44:35.956 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:44:35.956 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:44:35.956 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:44:35.956 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:44:35.956 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:44:35.956 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:35 smithi005 ceph-mon[24793]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37 2024-03-22T05:44:35.956 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:35 smithi005 ceph-mon[29523]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:44:35.956 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:44:35.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:44:35.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:44:35.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:44:35.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:44:35.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:44:35.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:44:35.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:44:35.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:44:35.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:44:35.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:44:35.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:44:35.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:44:35.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:44:35.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:44:35.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:44:35.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:44:35.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:44:35.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:44:35.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:44:35.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:44:35.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:44:35.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:44:35.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:44:35.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:44:35.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:44:35.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:44:35.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:44:35.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: return future.result(timeout) 2024-03-22T05:44:35.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:44:35.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.__get_result() 2024-03-22T05:44:35.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:44:35.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: raise self._exception 2024-03-22T05:44:35.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:44:35.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:44:35.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:44:35.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:44:35.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:44:35.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:44:35.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:44:35.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:44:35.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:44:35.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:44:35.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:44:35.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:44:35.960 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:44:35.960 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:44:35.960 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:44:35.960 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:44:35.960 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:44:35.960 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:44:35.960 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:44:35.960 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:44:35.960 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:44:35.960 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:44:35.960 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:44:35.960 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:44:35.960 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:44:35.960 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:35 smithi005 ceph-mon[29523]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37 2024-03-22T05:44:36.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:35 smithi186 ceph-mon[67903]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:44:36.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:44:36.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:44:36.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:44:36.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:44:36.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:44:36.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:44:36.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:44:36.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:44:36.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:44:36.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:44:36.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:44:36.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:44:36.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:44:36.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:44:36.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:44:36.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:44:36.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:44:36.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:44:36.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:44:36.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:44:36.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:44:36.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:44:36.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:44:36.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:44:36.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:44:36.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:44:36.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:44:36.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:44:36.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: return future.result(timeout) 2024-03-22T05:44:36.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:44:36.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.__get_result() 2024-03-22T05:44:36.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:44:36.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: raise self._exception 2024-03-22T05:44:36.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:44:36.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:44:36.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:44:36.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:44:36.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:44:36.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:44:36.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:44:36.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:44:36.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:44:36.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:44:36.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:44:36.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:44:36.048 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:44:36.048 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:44:36.048 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:44:36.048 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:44:36.048 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:44:36.048 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:44:36.048 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:44:36.048 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:44:36.048 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:44:36.048 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:44:36.048 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:44:36.048 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:44:36.048 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:44:36.048 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:35 smithi186 ceph-mon[67903]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37 2024-03-22T05:44:36.267 INFO:teuthology.orchestra.run.smithi005.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T05:44:36.267 INFO:teuthology.orchestra.run.smithi005.stdout:alertmanager.a smithi005 *:9093,9094 running (2m) 2m ago 24m 17.1M - 0.25.0 c8568f914cd2 ee99dfc435e8 2024-03-22T05:44:36.267 INFO:teuthology.orchestra.run.smithi005.stdout:grafana.a smithi186 *:3000 running (114s) 40s ago 23m 75.7M - 9.4.12 2bacad6d85d8 16fd57dbbd71 2024-03-22T05:44:36.267 INFO:teuthology.orchestra.run.smithi005.stdout:iscsi.foo.smithi005.mcdwbp smithi005 unknown 2m ago 23m 58.9M - 3.5 e1d6a67b021e 5215e03bf81b 2024-03-22T05:44:36.267 INFO:teuthology.orchestra.run.smithi005.stdout:mgr.x smithi186 *:8443,9283,8765 running (3m) 40s ago 28m 450M - 19.0.0-2299-gea01217f aa461e7c5e7f c556421a73f3 2024-03-22T05:44:36.267 INFO:teuthology.orchestra.run.smithi005.stdout:mgr.y smithi005 *:8443,9283,8765 running (13m) 2m ago 30m 503M - 19.0.0-2299-gea01217f aa461e7c5e7f 62a813eebc0d 2024-03-22T05:44:36.267 INFO:teuthology.orchestra.run.smithi005.stdout:mon.a smithi005 running (30m) 2m ago 30m 78.5M 2048M 17.2.0 e1d6a67b021e 38e0bba4bb03 2024-03-22T05:44:36.267 INFO:teuthology.orchestra.run.smithi005.stdout:mon.b smithi186 running (45s) 40s ago 28m 20.9M 2048M 19.0.0-2299-gea01217f aa461e7c5e7f c1798feac15b 2024-03-22T05:44:36.267 INFO:teuthology.orchestra.run.smithi005.stdout:mon.c smithi005 running (28m) 2m ago 28m 58.8M 2048M 17.2.0 e1d6a67b021e 00ec24d38612 2024-03-22T05:44:36.268 INFO:teuthology.orchestra.run.smithi005.stdout:node-exporter.a smithi005 *:9100 running (2m) 2m ago 24m 8786k - 1.5.0 0da6a335fe13 f635cd72b9d3 2024-03-22T05:44:36.268 INFO:teuthology.orchestra.run.smithi005.stdout:node-exporter.b smithi186 *:9100 running (2m) 40s ago 24m 14.5M - 1.5.0 0da6a335fe13 d7e0b3d20191 2024-03-22T05:44:36.268 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0 smithi005 running (28m) 2m ago 28m 53.5M 2993M 17.2.0 e1d6a67b021e ef5c7053dd37 2024-03-22T05:44:36.268 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1 smithi005 running (27m) 2m ago 27m 56.5M 2993M 17.2.0 e1d6a67b021e a1e7ee1017b6 2024-03-22T05:44:36.268 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2 smithi005 running (27m) 2m ago 27m 48.5M 2993M 17.2.0 e1d6a67b021e 462ea63ad0e5 2024-03-22T05:44:36.268 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3 smithi005 running (27m) 2m ago 27m 56.5M 2993M 17.2.0 e1d6a67b021e b25432a7239b 2024-03-22T05:44:36.268 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4 smithi186 running (26m) 40s ago 26m 56.4M 3248M 17.2.0 e1d6a67b021e cd963671f340 2024-03-22T05:44:36.268 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5 smithi186 running (26m) 40s ago 26m 51.6M 3248M 17.2.0 e1d6a67b021e f98c87f1a2cd 2024-03-22T05:44:36.268 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6 smithi186 running (25m) 40s ago 25m 52.0M 3248M 17.2.0 e1d6a67b021e 704ac49c68c4 2024-03-22T05:44:36.268 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7 smithi186 running (25m) 40s ago 25m 54.4M 3248M 17.2.0 e1d6a67b021e f415b90edf63 2024-03-22T05:44:36.268 INFO:teuthology.orchestra.run.smithi005.stdout:prometheus.a smithi186 *:9095 running (2m) 40s ago 24m 66.2M - 2.43.0 a07b618ecd1d a5a043588d6d 2024-03-22T05:44:36.268 INFO:teuthology.orchestra.run.smithi005.stdout:rgw.foo.smithi005.exdhqu smithi005 *:8000 running (23m) 2m ago 23m 78.8M - 17.2.0 e1d6a67b021e 6bb27fe11553 2024-03-22T05:44:36.269 INFO:teuthology.orchestra.run.smithi005.stdout:rgw.foo.smithi186.mcgodk smithi186 *:8000 running (23m) 40s ago 23m 77.9M - 17.2.0 e1d6a67b021e 19b5c20a34f7 2024-03-22T05:44:36.693 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-03-22T05:44:36.693 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": { 2024-03-22T05:44:36.693 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-03-22T05:44:36.693 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev)": 1 2024-03-22T05:44:36.693 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:44:36.694 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr": { 2024-03-22T05:44:36.694 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev)": 2 2024-03-22T05:44:36.694 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:44:36.694 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": { 2024-03-22T05:44:36.694 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-22T05:44:36.694 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:44:36.694 INFO:teuthology.orchestra.run.smithi005.stdout: "mds": {}, 2024-03-22T05:44:36.694 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw": { 2024-03-22T05:44:36.694 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-22T05:44:36.694 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:44:36.694 INFO:teuthology.orchestra.run.smithi005.stdout: "overall": { 2024-03-22T05:44:36.695 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 12, 2024-03-22T05:44:36.695 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev)": 3 2024-03-22T05:44:36.695 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-03-22T05:44:36.695 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-03-22T05:44:36.909 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:36 smithi005 ceph-mon[29523]: from='client.15678 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:44:36.909 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:36 smithi005 ceph-mon[29523]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:44:36.909 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:36 smithi005 ceph-mon[29523]: from='client.? 172.21.15.5:0/2287238494' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:44:36.909 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:36 smithi005 ceph-mon[24793]: from='client.15678 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:44:36.909 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:36 smithi005 ceph-mon[24793]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:44:36.909 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:36 smithi005 ceph-mon[24793]: from='client.? 172.21.15.5:0/2287238494' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:44:37.042 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-03-22T05:44:37.042 INFO:teuthology.orchestra.run.smithi005.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9", 2024-03-22T05:44:37.042 INFO:teuthology.orchestra.run.smithi005.stdout: "in_progress": true, 2024-03-22T05:44:37.043 INFO:teuthology.orchestra.run.smithi005.stdout: "which": "Upgrading daemons of type(s) mon on host(s) smithi005", 2024-03-22T05:44:37.043 INFO:teuthology.orchestra.run.smithi005.stdout: "services_complete": [], 2024-03-22T05:44:37.043 INFO:teuthology.orchestra.run.smithi005.stdout: "progress": "0/2 daemons upgraded", 2024-03-22T05:44:37.043 INFO:teuthology.orchestra.run.smithi005.stdout: "message": "Currently upgrading mon daemons", 2024-03-22T05:44:37.043 INFO:teuthology.orchestra.run.smithi005.stdout: "is_paused": false 2024-03-22T05:44:37.043 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-03-22T05:44:37.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:36 smithi186 ceph-mon[67903]: from='client.15678 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:44:37.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:36 smithi186 ceph-mon[67903]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:44:37.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:36 smithi186 ceph-mon[67903]: from='client.? 172.21.15.5:0/2287238494' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:44:37.202 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:36 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:44:36] "GET /metrics HTTP/1.1" 200 37995 "" "Prometheus/2.43.0" 2024-03-22T05:44:37.881 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:37 smithi005 ceph-mon[24793]: from='client.15684 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:44:37.881 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:37 smithi005 ceph-mon[24793]: from='client.15690 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:44:37.881 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:37 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:44:37.881 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:37 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:44:37.881 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:37 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:44:37.881 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:37 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:44:37.881 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:37 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-22T05:44:37.881 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:37 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-03-22T05:44:37.881 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:37 smithi005 ceph-mon[29523]: from='client.15684 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:44:37.881 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:37 smithi005 ceph-mon[29523]: from='client.15690 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:44:37.882 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:37 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:44:37.882 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:37 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:44:37.882 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:37 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:44:37.882 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:37 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:44:37.882 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:37 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-22T05:44:37.882 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:37 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-03-22T05:44:38.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:37 smithi186 ceph-mon[67903]: from='client.15684 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:44:38.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:37 smithi186 ceph-mon[67903]: from='client.15690 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:44:38.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:37 smithi186 ceph-mon[67903]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:44:38.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:37 smithi186 ceph-mon[67903]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:44:38.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:37 smithi186 ceph-mon[67903]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:44:38.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:37 smithi186 ceph-mon[67903]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:44:38.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:37 smithi186 ceph-mon[67903]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-22T05:44:38.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:37 smithi186 ceph-mon[67903]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-03-22T05:44:38.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:38 smithi005 ceph-mon[24793]: Upgrade: Target is version 19.0.0-2299-gea01217f (squid) 2024-03-22T05:44:38.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:38 smithi005 ceph-mon[24793]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, digests ['quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9'] 2024-03-22T05:44:38.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:38 smithi005 ceph-mon[24793]: Upgrade: Setting container_image for all mgr 2024-03-22T05:44:38.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:38 smithi005 ceph-mon[24793]: Upgrade: It appears safe to stop mon.a 2024-03-22T05:44:38.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:38 smithi005 ceph-mon[24793]: from='client.25603 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:44:38.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:38 smithi005 ceph-mon[24793]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:44:38.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:38 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:44:38.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:38 smithi005 ceph-mon[24793]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:44:38.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:38 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-22T05:44:38.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:38 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-22T05:44:38.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:38 smithi005 ceph-mon[24793]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:44:38.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:38 smithi005 ceph-mon[29523]: Upgrade: Target is version 19.0.0-2299-gea01217f (squid) 2024-03-22T05:44:38.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:38 smithi005 ceph-mon[29523]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, digests ['quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9'] 2024-03-22T05:44:38.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:38 smithi005 ceph-mon[29523]: Upgrade: Setting container_image for all mgr 2024-03-22T05:44:38.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:38 smithi005 ceph-mon[29523]: Upgrade: It appears safe to stop mon.a 2024-03-22T05:44:38.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:38 smithi005 ceph-mon[29523]: from='client.25603 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:44:38.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:38 smithi005 ceph-mon[29523]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:44:38.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:38 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:44:38.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:38 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:44:38.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:38 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-22T05:44:38.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:38 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-22T05:44:38.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:38 smithi005 ceph-mon[29523]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:44:39.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:38 smithi186 ceph-mon[67903]: Upgrade: Target is version 19.0.0-2299-gea01217f (squid) 2024-03-22T05:44:39.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:38 smithi186 ceph-mon[67903]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, digests ['quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9'] 2024-03-22T05:44:39.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:38 smithi186 ceph-mon[67903]: Upgrade: Setting container_image for all mgr 2024-03-22T05:44:39.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:38 smithi186 ceph-mon[67903]: Upgrade: It appears safe to stop mon.a 2024-03-22T05:44:39.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:38 smithi186 ceph-mon[67903]: from='client.25603 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:44:39.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:38 smithi186 ceph-mon[67903]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:44:39.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:38 smithi186 ceph-mon[67903]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:44:39.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:38 smithi186 ceph-mon[67903]: from='mgr.15351 ' entity='mgr.y' 2024-03-22T05:44:39.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:38 smithi186 ceph-mon[67903]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-22T05:44:39.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:38 smithi186 ceph-mon[67903]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-22T05:44:39.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:38 smithi186 ceph-mon[67903]: from='mgr.15351 172.21.15.5:0/1143295871' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:44:40.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:39 smithi186 ceph-mon[67903]: Upgrade: Updating mon.a 2024-03-22T05:44:40.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:39 smithi186 ceph-mon[67903]: Deploying daemon mon.a on smithi005 2024-03-22T05:44:40.088 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:39 smithi005 ceph-mon[24793]: Upgrade: Updating mon.a 2024-03-22T05:44:40.088 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:39 smithi005 ceph-mon[24793]: Deploying daemon mon.a on smithi005 2024-03-22T05:44:40.088 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:39 smithi005 ceph-mon[29523]: Upgrade: Updating mon.a 2024-03-22T05:44:40.088 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:39 smithi005 ceph-mon[29523]: Deploying daemon mon.a on smithi005 2024-03-22T05:44:40.637 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:40 smithi005 systemd[1]: Stopping Ceph mon.a for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:44:40.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mon-a[24789]: 2024-03-22T05:44:40.636+0000 7eff178cc700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T05:44:40.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mon-a[24789]: 2024-03-22T05:44:40.636+0000 7eff178cc700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-22T05:44:40.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:40 smithi005 podman[84539]: 2024-03-22 05:44:40.709011386 +0000 UTC m=+0.185183399 container died 38e0bba4bb0324623e0b8652e83145cb7b3fa3105e16aff7dcc8c740ed0838d3 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mon-a, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, RELEASE=HEAD, vcs-type=git, release=754, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, com.redhat.component=centos-stream-container, distribution-scope=public, io.openshift.expose-services=, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-03-22T05:44:41.359 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:41 smithi005 podman[84539]: 2024-03-22 05:44:41.070423053 +0000 UTC m=+0.546595065 container cleanup 38e0bba4bb0324623e0b8652e83145cb7b3fa3105e16aff7dcc8c740ed0838d3 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mon-a, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, io.openshift.expose-services=, architecture=x86_64, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, name=centos-stream, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, ceph=True, vcs-type=git, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-03-22T05:44:41.359 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:41 smithi005 bash[84539]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mon-a 2024-03-22T05:44:41.684 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:41 smithi005 podman[84554]: 2024-03-22 05:44:41.360228265 +0000 UTC m=+0.648372739 container remove 38e0bba4bb0324623e0b8652e83145cb7b3fa3105e16aff7dcc8c740ed0838d3 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mon-a, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, version=8, com.redhat.component=centos-stream-container, GIT_CLEAN=True, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, release=754, GIT_BRANCH=HEAD, vcs-type=git, vendor=Red Hat, Inc., distribution-scope=public, io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-03-22T05:44:41.685 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:41 smithi005 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-22T05:44:41.685 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:41 smithi005 systemd[1]: Stopped Ceph mon.a for f59f36e2-e80a-11ee-95cd-87774f69a715. 2024-03-22T05:44:41.685 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:41 smithi005 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@mon.a.service: Consumed 25.996s CPU time. 2024-03-22T05:44:41.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:41 smithi005 systemd[1]: Starting Ceph mon.a for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:44:41.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:41 smithi005 podman[84685]: 2024-03-22 05:44:41.8980989 +0000 UTC m=+0.114525644 container create 111a2aa24dd942b914010b3779e02df03411258da7eb5f9e8c8139f5ca03e9ad (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mon-a, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, CEPH_POINT_RELEASE=) 2024-03-22T05:44:41.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:41 smithi005 podman[84685]: 2024-03-22 05:44:41.801075373 +0000 UTC m=+0.017502120 image pull aa461e7c5e7f322b169c789b389d594b994ea7ffcbc9e24e563a3ec9b69bc3c1 quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9 2024-03-22T05:44:42.419 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 podman[84685]: 2024-03-22 05:44:42.1346733 +0000 UTC m=+0.351100058 container init 111a2aa24dd942b914010b3779e02df03411258da7eb5f9e8c8139f5ca03e9ad (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217) 2024-03-22T05:44:42.419 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 podman[84685]: 2024-03-22 05:44:42.138017771 +0000 UTC m=+0.354444515 container start 111a2aa24dd942b914010b3779e02df03411258da7eb5f9e8c8139f5ca03e9ad (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mon-a, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-22T05:44:42.419 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: set uid:gid to 167:167 (ceph:ceph) 2024-03-22T05:44:42.420 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev), process ceph-mon, pid 2 2024-03-22T05:44:42.420 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: pidfile_write: ignore empty --pid-file 2024-03-22T05:44:42.420 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: load: jerasure load: lrc 2024-03-22T05:44:42.420 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: RocksDB version: 7.9.2 2024-03-22T05:44:42.420 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Git sha 0 2024-03-22T05:44:42.420 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Compile date 2024-03-21 23:09:57 2024-03-22T05:44:42.420 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: DB SUMMARY 2024-03-22T05:44:42.420 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: DB Session ID: YOMY6RW87JBHFUR9987D 2024-03-22T05:44:42.420 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: CURRENT file: CURRENT 2024-03-22T05:44:42.420 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: IDENTITY file: IDENTITY 2024-03-22T05:44:42.420 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: MANIFEST file: MANIFEST-000015 size: 2889 Bytes 2024-03-22T05:44:42.420 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000060.sst 2024-03-22T05:44:42.420 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000058.log size: 2601556 ; 2024-03-22T05:44:42.420 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.error_if_exists: 0 2024-03-22T05:44:42.421 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.create_if_missing: 0 2024-03-22T05:44:42.421 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.paranoid_checks: 1 2024-03-22T05:44:42.421 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-22T05:44:42.421 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-22T05:44:42.421 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-22T05:44:42.421 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.env: 0x55f79a97db20 2024-03-22T05:44:42.421 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.fs: PosixFileSystem 2024-03-22T05:44:42.421 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.info_log: 0x55f79cb2d920 2024-03-22T05:44:42.421 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.max_file_opening_threads: 16 2024-03-22T05:44:42.421 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.statistics: (nil) 2024-03-22T05:44:42.421 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.use_fsync: 0 2024-03-22T05:44:42.421 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.max_log_file_size: 0 2024-03-22T05:44:42.421 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-22T05:44:42.421 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-22T05:44:42.421 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.keep_log_file_num: 1000 2024-03-22T05:44:42.422 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.recycle_log_file_num: 0 2024-03-22T05:44:42.422 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.allow_fallocate: 1 2024-03-22T05:44:42.422 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.allow_mmap_reads: 0 2024-03-22T05:44:42.422 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.allow_mmap_writes: 0 2024-03-22T05:44:42.422 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.use_direct_reads: 0 2024-03-22T05:44:42.422 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-22T05:44:42.422 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.create_missing_column_families: 0 2024-03-22T05:44:42.422 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.db_log_dir: 2024-03-22T05:44:42.422 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.wal_dir: 2024-03-22T05:44:42.422 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-22T05:44:42.422 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-22T05:44:42.422 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-22T05:44:42.422 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-22T05:44:42.422 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-22T05:44:42.423 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-22T05:44:42.423 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.advise_random_on_open: 1 2024-03-22T05:44:42.423 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.db_write_buffer_size: 0 2024-03-22T05:44:42.423 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.write_buffer_manager: 0x55f79cafe1e0 2024-03-22T05:44:42.423 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-22T05:44:42.423 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-22T05:44:42.423 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-22T05:44:42.423 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.rate_limiter: (nil) 2024-03-22T05:44:42.423 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-22T05:44:42.423 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.wal_recovery_mode: 2 2024-03-22T05:44:42.423 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.enable_thread_tracking: 0 2024-03-22T05:44:42.423 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.enable_pipelined_write: 0 2024-03-22T05:44:42.423 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.unordered_write: 0 2024-03-22T05:44:42.423 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-22T05:44:42.424 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-22T05:44:42.424 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-22T05:44:42.424 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-22T05:44:42.424 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.row_cache: None 2024-03-22T05:44:42.424 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.wal_filter: None 2024-03-22T05:44:42.425 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-22T05:44:42.425 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.allow_ingest_behind: 0 2024-03-22T05:44:42.425 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.two_write_queues: 0 2024-03-22T05:44:42.425 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.manual_wal_flush: 0 2024-03-22T05:44:42.425 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.wal_compression: 0 2024-03-22T05:44:42.425 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.atomic_flush: 0 2024-03-22T05:44:42.425 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-22T05:44:42.425 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-22T05:44:42.426 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-22T05:44:42.426 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.log_readahead_size: 0 2024-03-22T05:44:42.426 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-22T05:44:42.426 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.best_efforts_recovery: 0 2024-03-22T05:44:42.426 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-22T05:44:42.426 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-22T05:44:42.426 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.allow_data_in_errors: 0 2024-03-22T05:44:42.426 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.db_host_id: __hostname__ 2024-03-22T05:44:42.426 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-22T05:44:42.426 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.max_background_jobs: 2 2024-03-22T05:44:42.426 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.max_background_compactions: -1 2024-03-22T05:44:42.426 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.max_subcompactions: 1 2024-03-22T05:44:42.426 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-22T05:44:42.427 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-22T05:44:42.427 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-22T05:44:42.427 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.max_total_wal_size: 0 2024-03-22T05:44:42.427 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-22T05:44:42.427 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-22T05:44:42.427 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-22T05:44:42.427 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-22T05:44:42.427 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.max_open_files: -1 2024-03-22T05:44:42.427 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.bytes_per_sync: 0 2024-03-22T05:44:42.427 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-22T05:44:42.427 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-22T05:44:42.427 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.compaction_readahead_size: 0 2024-03-22T05:44:42.427 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.max_background_flushes: -1 2024-03-22T05:44:42.427 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Compression algorithms supported: 2024-03-22T05:44:42.427 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-22T05:44:42.428 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: kZSTD supported: 0 2024-03-22T05:44:42.428 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: kXpressCompression supported: 0 2024-03-22T05:44:42.428 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: kLZ4HCCompression supported: 1 2024-03-22T05:44:42.428 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: kZlibCompression supported: 1 2024-03-22T05:44:42.428 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: kSnappyCompression supported: 1 2024-03-22T05:44:42.428 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: kLZ4Compression supported: 1 2024-03-22T05:44:42.428 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: kBZip2Compression supported: 0 2024-03-22T05:44:42.428 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-22T05:44:42.428 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-22T05:44:42.428 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 2024-03-22T05:44:42.428 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-22T05:44:42.428 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-22T05:44:42.428 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.merge_operator: 2024-03-22T05:44:42.428 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.compaction_filter: None 2024-03-22T05:44:42.429 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.compaction_filter_factory: None 2024-03-22T05:44:42.429 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.sst_partitioner_factory: None 2024-03-22T05:44:42.429 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-22T05:44:42.429 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-22T05:44:42.429 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55f79cb2da20) 2024-03-22T05:44:42.429 INFO:journalctl@ceph.mon.a.smithi005.stdout: cache_index_and_filter_blocks: 1 2024-03-22T05:44:42.429 INFO:journalctl@ceph.mon.a.smithi005.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-22T05:44:42.429 INFO:journalctl@ceph.mon.a.smithi005.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-22T05:44:42.429 INFO:journalctl@ceph.mon.a.smithi005.stdout: pin_top_level_index_and_filter: 1 2024-03-22T05:44:42.429 INFO:journalctl@ceph.mon.a.smithi005.stdout: index_type: 0 2024-03-22T05:44:42.429 INFO:journalctl@ceph.mon.a.smithi005.stdout: data_block_index_type: 0 2024-03-22T05:44:42.429 INFO:journalctl@ceph.mon.a.smithi005.stdout: index_shortening: 1 2024-03-22T05:44:42.429 INFO:journalctl@ceph.mon.a.smithi005.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-22T05:44:42.429 INFO:journalctl@ceph.mon.a.smithi005.stdout: checksum: 4 2024-03-22T05:44:42.430 INFO:journalctl@ceph.mon.a.smithi005.stdout: no_block_cache: 0 2024-03-22T05:44:42.430 INFO:journalctl@ceph.mon.a.smithi005.stdout: block_cache: 0x55f79cb31090 2024-03-22T05:44:42.430 INFO:journalctl@ceph.mon.a.smithi005.stdout: block_cache_name: BinnedLRUCache 2024-03-22T05:44:42.430 INFO:journalctl@ceph.mon.a.smithi005.stdout: block_cache_options: 2024-03-22T05:44:42.430 INFO:journalctl@ceph.mon.a.smithi005.stdout: capacity : 536870912 2024-03-22T05:44:42.430 INFO:journalctl@ceph.mon.a.smithi005.stdout: num_shard_bits : 4 2024-03-22T05:44:42.431 INFO:journalctl@ceph.mon.a.smithi005.stdout: strict_capacity_limit : 0 2024-03-22T05:44:42.431 INFO:journalctl@ceph.mon.a.smithi005.stdout: high_pri_pool_ratio: 0.000 2024-03-22T05:44:42.431 INFO:journalctl@ceph.mon.a.smithi005.stdout: block_cache_compressed: (nil) 2024-03-22T05:44:42.431 INFO:journalctl@ceph.mon.a.smithi005.stdout: persistent_cache: (nil) 2024-03-22T05:44:42.431 INFO:journalctl@ceph.mon.a.smithi005.stdout: block_size: 4096 2024-03-22T05:44:42.432 INFO:journalctl@ceph.mon.a.smithi005.stdout: block_size_deviation: 10 2024-03-22T05:44:42.432 INFO:journalctl@ceph.mon.a.smithi005.stdout: block_restart_interval: 16 2024-03-22T05:44:42.432 INFO:journalctl@ceph.mon.a.smithi005.stdout: index_block_restart_interval: 1 2024-03-22T05:44:42.432 INFO:journalctl@ceph.mon.a.smithi005.stdout: metadata_block_size: 4096 2024-03-22T05:44:42.432 INFO:journalctl@ceph.mon.a.smithi005.stdout: partition_filters: 0 2024-03-22T05:44:42.432 INFO:journalctl@ceph.mon.a.smithi005.stdout: use_delta_encoding: 1 2024-03-22T05:44:42.432 INFO:journalctl@ceph.mon.a.smithi005.stdout: filter_policy: bloomfilter 2024-03-22T05:44:42.432 INFO:journalctl@ceph.mon.a.smithi005.stdout: whole_key_filtering: 1 2024-03-22T05:44:42.432 INFO:journalctl@ceph.mon.a.smithi005.stdout: verify_compression: 0 2024-03-22T05:44:42.432 INFO:journalctl@ceph.mon.a.smithi005.stdout: read_amp_bytes_per_bit: 0 2024-03-22T05:44:42.432 INFO:journalctl@ceph.mon.a.smithi005.stdout: format_version: 5 2024-03-22T05:44:42.433 INFO:journalctl@ceph.mon.a.smithi005.stdout: enable_index_compression: 1 2024-03-22T05:44:42.433 INFO:journalctl@ceph.mon.a.smithi005.stdout: block_align: 0 2024-03-22T05:44:42.433 INFO:journalctl@ceph.mon.a.smithi005.stdout: max_auto_readahead_size: 262144 2024-03-22T05:44:42.433 INFO:journalctl@ceph.mon.a.smithi005.stdout: prepopulate_block_cache: 0 2024-03-22T05:44:42.433 INFO:journalctl@ceph.mon.a.smithi005.stdout: initial_auto_readahead_size: 8192 2024-03-22T05:44:42.433 INFO:journalctl@ceph.mon.a.smithi005.stdout: num_file_reads_for_auto_readahead: 2 2024-03-22T05:44:42.433 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.write_buffer_size: 33554432 2024-03-22T05:44:42.433 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.max_write_buffer_number: 2 2024-03-22T05:44:42.433 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.compression: NoCompression 2024-03-22T05:44:42.433 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.bottommost_compression: Disabled 2024-03-22T05:44:42.433 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.prefix_extractor: nullptr 2024-03-22T05:44:42.434 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-22T05:44:42.434 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.num_levels: 7 2024-03-22T05:44:42.434 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-22T05:44:42.434 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-22T05:44:42.434 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-22T05:44:42.434 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-22T05:44:42.434 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-22T05:44:42.434 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-22T05:44:42.434 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-22T05:44:42.434 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-22T05:44:42.434 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-22T05:44:42.434 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-22T05:44:42.434 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-22T05:44:42.434 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-22T05:44:42.434 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-22T05:44:42.435 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.compression_opts.level: 32767 2024-03-22T05:44:42.435 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.compression_opts.strategy: 0 2024-03-22T05:44:42.435 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-22T05:44:42.435 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-22T05:44:42.435 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-22T05:44:42.435 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-22T05:44:42.435 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.compression_opts.enabled: false 2024-03-22T05:44:42.435 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-22T05:44:42.435 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-22T05:44:42.435 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-22T05:44:42.436 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-22T05:44:42.436 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.target_file_size_base: 67108864 2024-03-22T05:44:42.436 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-22T05:44:42.436 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-22T05:44:42.436 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-22T05:44:42.436 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-22T05:44:42.436 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-22T05:44:42.436 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-22T05:44:42.436 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-22T05:44:42.436 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-22T05:44:42.436 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-22T05:44:42.436 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-22T05:44:42.437 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-22T05:44:42.437 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-22T05:44:42.437 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-22T05:44:42.437 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-22T05:44:42.437 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.arena_block_size: 1048576 2024-03-22T05:44:42.437 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-22T05:44:42.437 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-22T05:44:42.437 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.disable_auto_compactions: 0 2024-03-22T05:44:42.437 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-22T05:44:42.437 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-22T05:44:42.437 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-22T05:44:42.437 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-22T05:44:42.437 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-22T05:44:42.437 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-22T05:44:42.437 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-22T05:44:42.438 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-22T05:44:42.438 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-22T05:44:42.438 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-22T05:44:42.438 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.table_properties_collectors: 2024-03-22T05:44:42.438 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.inplace_update_support: 0 2024-03-22T05:44:42.438 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-22T05:44:42.438 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-22T05:44:42.438 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-22T05:44:42.438 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-22T05:44:42.438 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.bloom_locality: 0 2024-03-22T05:44:42.438 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.max_successive_merges: 0 2024-03-22T05:44:42.438 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-22T05:44:42.438 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.paranoid_file_checks: 0 2024-03-22T05:44:42.438 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.force_consistency_checks: 1 2024-03-22T05:44:42.438 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.report_bg_io_stats: 0 2024-03-22T05:44:42.439 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.ttl: 2592000 2024-03-22T05:44:42.439 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-22T05:44:42.439 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-22T05:44:42.439 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-22T05:44:42.439 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.enable_blob_files: false 2024-03-22T05:44:42.439 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.min_blob_size: 0 2024-03-22T05:44:42.439 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.blob_file_size: 268435456 2024-03-22T05:44:42.439 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-22T05:44:42.439 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-22T05:44:42.439 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-22T05:44:42.439 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-22T05:44:42.439 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-22T05:44:42.439 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.blob_file_starting_level: 0 2024-03-22T05:44:42.439 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-22T05:44:42.439 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 60.sst 2024-03-22T05:44:42.440 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-22T05:44:42.440 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 succeeded,manifest_file_number is 15, next_file_number is 62, last_sequence is 26813, log_number is 58,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-22T05:44:42.440 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 58 2024-03-22T05:44:42.440 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 5e89885b-5fdf-4fe0-8169-5b6a3e97d4f9 2024-03-22T05:44:42.440 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711086282167642, "job": 1, "event": "recovery_started", "wal_files": [58]} 2024-03-22T05:44:42.440 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #58 mode 2 2024-03-22T05:44:42.440 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711086282176357, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 63, "file_size": 2447010, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 26814, "largest_seqno": 29092, "table_properties": {"data_size": 2437642, "index_size": 5724, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 2629, "raw_key_size": 25734, "raw_average_key_size": 24, "raw_value_size": 2416530, "raw_average_value_size": 2341, "num_data_blocks": 253, "num_entries": 1032, "num_filter_entries": 1032, "num_deletions": 8, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711086282, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "5e89885b-5fdf-4fe0-8169-5b6a3e97d4f9", "db_session_id": "YOMY6RW87JBHFUR9987D", "orig_file_number": 63, "seqno_to_time_mapping": "N/A"}} 2024-03-22T05:44:42.440 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711086282176439, "job": 1, "event": "recovery_finished"} 2024-03-22T05:44:42.440 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: [db/version_set.cc:5047] Creating manifest 65 2024-03-22T05:44:42.440 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-22T05:44:42.440 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000058.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-22T05:44:42.440 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55f79cc20000 2024-03-22T05:44:42.440 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: DB pointer 0x55f79cc0a000 2024-03-22T05:44:42.440 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-22T05:44:42.441 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-22T05:44:42.441 INFO:journalctl@ceph.mon.a.smithi005.stdout: ** DB Stats ** 2024-03-22T05:44:42.441 INFO:journalctl@ceph.mon.a.smithi005.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-22T05:44:42.441 INFO:journalctl@ceph.mon.a.smithi005.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-22T05:44:42.441 INFO:journalctl@ceph.mon.a.smithi005.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-22T05:44:42.441 INFO:journalctl@ceph.mon.a.smithi005.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-22T05:44:42.441 INFO:journalctl@ceph.mon.a.smithi005.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-22T05:44:42.441 INFO:journalctl@ceph.mon.a.smithi005.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-22T05:44:42.441 INFO:journalctl@ceph.mon.a.smithi005.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-22T05:44:42.441 INFO:journalctl@ceph.mon.a.smithi005.stdout: 2024-03-22T05:44:42.441 INFO:journalctl@ceph.mon.a.smithi005.stdout: ** Compaction Stats [default] ** 2024-03-22T05:44:42.441 INFO:journalctl@ceph.mon.a.smithi005.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-22T05:44:42.441 INFO:journalctl@ceph.mon.a.smithi005.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-22T05:44:42.441 INFO:journalctl@ceph.mon.a.smithi005.stdout: L0 1/0 2.33 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 535.5 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-03-22T05:44:42.441 INFO:journalctl@ceph.mon.a.smithi005.stdout: L6 1/0 8.91 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-03-22T05:44:42.442 INFO:journalctl@ceph.mon.a.smithi005.stdout: Sum 2/0 11.24 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 535.5 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-03-22T05:44:42.442 INFO:journalctl@ceph.mon.a.smithi005.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 535.5 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-03-22T05:44:42.442 INFO:journalctl@ceph.mon.a.smithi005.stdout: 2024-03-22T05:44:42.442 INFO:journalctl@ceph.mon.a.smithi005.stdout: ** Compaction Stats [default] ** 2024-03-22T05:44:42.442 INFO:journalctl@ceph.mon.a.smithi005.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-22T05:44:42.442 INFO:journalctl@ceph.mon.a.smithi005.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-22T05:44:42.442 INFO:journalctl@ceph.mon.a.smithi005.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 535.5 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-03-22T05:44:42.442 INFO:journalctl@ceph.mon.a.smithi005.stdout: 2024-03-22T05:44:42.442 INFO:journalctl@ceph.mon.a.smithi005.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-22T05:44:42.442 INFO:journalctl@ceph.mon.a.smithi005.stdout: 2024-03-22T05:44:42.443 INFO:journalctl@ceph.mon.a.smithi005.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-22T05:44:42.443 INFO:journalctl@ceph.mon.a.smithi005.stdout: Flush(GB): cumulative 0.002, interval 0.002 2024-03-22T05:44:42.443 INFO:journalctl@ceph.mon.a.smithi005.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-22T05:44:42.443 INFO:journalctl@ceph.mon.a.smithi005.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-22T05:44:42.443 INFO:journalctl@ceph.mon.a.smithi005.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-22T05:44:42.443 INFO:journalctl@ceph.mon.a.smithi005.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-22T05:44:42.443 INFO:journalctl@ceph.mon.a.smithi005.stdout: Cumulative compaction: 0.00 GB write, 210.05 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-22T05:44:42.443 INFO:journalctl@ceph.mon.a.smithi005.stdout: Interval compaction: 0.00 GB write, 210.05 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-22T05:44:42.443 INFO:journalctl@ceph.mon.a.smithi005.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-22T05:44:42.443 INFO:journalctl@ceph.mon.a.smithi005.stdout: Block cache BinnedLRUCache@0x55f79cb31090#2 capacity: 512.00 MB usage: 8.86 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-03-22T05:44:42.443 INFO:journalctl@ceph.mon.a.smithi005.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,2.78 KB,0.000530481%) IndexBlock(1,6.08 KB,0.00115931%) Misc(1,0.00 KB,0%) 2024-03-22T05:44:42.443 INFO:journalctl@ceph.mon.a.smithi005.stdout: 2024-03-22T05:44:42.443 INFO:journalctl@ceph.mon.a.smithi005.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-22T05:44:42.443 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: starting mon.a rank 0 at public addrs [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] at bind addrs [v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid f59f36e2-e80a-11ee-95cd-87774f69a715 2024-03-22T05:44:42.443 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: mon.a@-1(???) e3 preinit fsid f59f36e2-e80a-11ee-95cd-87774f69a715 2024-03-22T05:44:42.444 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: mon.a@-1(???).mds e1 new map 2024-03-22T05:44:42.444 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: mon.a@-1(???).mds e1 print_map 2024-03-22T05:44:42.444 INFO:journalctl@ceph.mon.a.smithi005.stdout: e1 2024-03-22T05:44:42.444 INFO:journalctl@ceph.mon.a.smithi005.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-22T05:44:42.444 INFO:journalctl@ceph.mon.a.smithi005.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T05:44:42.444 INFO:journalctl@ceph.mon.a.smithi005.stdout: legacy client fscid: -1 2024-03-22T05:44:42.444 INFO:journalctl@ceph.mon.a.smithi005.stdout: 2024-03-22T05:44:42.444 INFO:journalctl@ceph.mon.a.smithi005.stdout: No filesystems configured 2024-03-22T05:44:42.444 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: mon.a@-1(???).osd e104 crush map has features 3314933000854323200, adjusting msgr requires 2024-03-22T05:44:42.444 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: mon.a@-1(???).osd e104 crush map has features 432629239337189376, adjusting msgr requires 2024-03-22T05:44:42.444 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: mon.a@-1(???).osd e104 crush map has features 432629239337189376, adjusting msgr requires 2024-03-22T05:44:42.444 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: mon.a@-1(???).osd e104 crush map has features 432629239337189376, adjusting msgr requires 2024-03-22T05:44:42.444 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 ceph-mon[84705]: mon.a@-1(???).paxosservice(auth 1..27) refresh upgraded, format 0 -> 3 2024-03-22T05:44:42.444 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 bash[84685]: 111a2aa24dd942b914010b3779e02df03411258da7eb5f9e8c8139f5ca03e9ad 2024-03-22T05:44:42.445 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:42 smithi005 systemd[1]: Started Ceph mon.a for f59f36e2-e80a-11ee-95cd-87774f69a715. 2024-03-22T05:44:45.544 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:44:45 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[61309]: ignoring --setuser ceph since I am not root 2024-03-22T05:44:45.544 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:44:45 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[61309]: ignoring --setgroup ceph since I am not root 2024-03-22T05:44:45.544 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:44:45 smithi186 ceph-mgr[61313]: -- 172.21.15.186:0/1482424579 <== mon.2 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x559ad799c5a0 con 0x559ad7970800 2024-03-22T05:44:45.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:45 smithi186 ceph-mon[67903]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:44:45.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:45 smithi186 ceph-mon[67903]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:44:45.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:45 smithi186 ceph-mon[67903]: mon.a calling monitor election 2024-03-22T05:44:45.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:45 smithi186 ceph-mon[67903]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:44:45.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:45 smithi186 ceph-mon[67903]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-22T05:44:45.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:45 smithi186 ceph-mon[67903]: monmap e3: 3 mons at {a=[v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0],b=[v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0],c=[v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-22T05:44:45.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:45 smithi186 ceph-mon[67903]: fsmap 2024-03-22T05:44:45.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:45 smithi186 ceph-mon[67903]: osdmap e104: 8 total, 8 up, 8 in 2024-03-22T05:44:45.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:45 smithi186 ceph-mon[67903]: mgrmap e43: y(active, since 3m), standbys: x 2024-03-22T05:44:45.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:45 smithi186 ceph-mon[67903]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-22T05:44:45.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:45 smithi186 ceph-mon[67903]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-22T05:44:45.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:45 smithi186 ceph-mon[67903]: daemon iscsi.foo.smithi005.mcdwbp on smithi005 is in unknown state 2024-03-22T05:44:45.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:45 smithi186 ceph-mon[67903]: from='mgr.15351 ' entity='' 2024-03-22T05:44:45.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:45 smithi186 ceph-mon[67903]: mgrmap e44: y(active, since 3m), standbys: x 2024-03-22T05:44:45.702 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:45 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ignoring --setuser ceph since I am not root 2024-03-22T05:44:45.702 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:45 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ignoring --setgroup ceph since I am not root 2024-03-22T05:44:45.702 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:45 smithi005 ceph-mgr[63233]: -- 172.21.15.5:0/2973259795 <== mon.2 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5638155f25a0 con 0x5638155c7000 2024-03-22T05:44:45.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:45 smithi005 ceph-mon[84705]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:44:45.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:45 smithi005 ceph-mon[84705]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:44:45.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:45 smithi005 ceph-mon[84705]: mon.a calling monitor election 2024-03-22T05:44:45.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:45 smithi005 ceph-mon[84705]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:44:45.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:45 smithi005 ceph-mon[84705]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-22T05:44:45.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:45 smithi005 ceph-mon[84705]: monmap e3: 3 mons at {a=[v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0],b=[v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0],c=[v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-22T05:44:45.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:45 smithi005 ceph-mon[84705]: fsmap 2024-03-22T05:44:45.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:45 smithi005 ceph-mon[84705]: osdmap e104: 8 total, 8 up, 8 in 2024-03-22T05:44:45.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:45 smithi005 ceph-mon[84705]: mgrmap e43: y(active, since 3m), standbys: x 2024-03-22T05:44:45.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:45 smithi005 ceph-mon[84705]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-22T05:44:45.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:45 smithi005 ceph-mon[84705]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-22T05:44:45.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:45 smithi005 ceph-mon[84705]: daemon iscsi.foo.smithi005.mcdwbp on smithi005 is in unknown state 2024-03-22T05:44:45.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:45 smithi005 ceph-mon[84705]: from='mgr.15351 ' entity='' 2024-03-22T05:44:45.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:45 smithi005 ceph-mon[84705]: mgrmap e44: y(active, since 3m), standbys: x 2024-03-22T05:44:45.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:45 smithi005 ceph-mon[29523]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:44:45.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:45 smithi005 ceph-mon[29523]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:44:45.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:45 smithi005 ceph-mon[29523]: mon.a calling monitor election 2024-03-22T05:44:45.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:45 smithi005 ceph-mon[29523]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:44:45.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:45 smithi005 ceph-mon[29523]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-22T05:44:45.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:45 smithi005 ceph-mon[29523]: monmap e3: 3 mons at {a=[v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0],b=[v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0],c=[v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-22T05:44:45.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:45 smithi005 ceph-mon[29523]: fsmap 2024-03-22T05:44:45.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:45 smithi005 ceph-mon[29523]: osdmap e104: 8 total, 8 up, 8 in 2024-03-22T05:44:45.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:45 smithi005 ceph-mon[29523]: mgrmap e43: y(active, since 3m), standbys: x 2024-03-22T05:44:45.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:45 smithi005 ceph-mon[29523]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-22T05:44:45.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:45 smithi005 ceph-mon[29523]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-22T05:44:45.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:45 smithi005 ceph-mon[29523]: daemon iscsi.foo.smithi005.mcdwbp on smithi005 is in unknown state 2024-03-22T05:44:45.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:45 smithi005 ceph-mon[29523]: from='mgr.15351 ' entity='' 2024-03-22T05:44:45.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:45 smithi005 ceph-mon[29523]: mgrmap e44: y(active, since 3m), standbys: x 2024-03-22T05:44:46.202 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:45 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:44:45.831+0000 7f7f883ce200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-22T05:44:46.202 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:45 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:44:45.928+0000 7f7f883ce200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-22T05:44:46.293 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:44:45 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[61309]: 2024-03-22T05:44:45.808+0000 7f0e20278200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-22T05:44:46.293 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:44:45 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[61309]: 2024-03-22T05:44:45.902+0000 7f0e20278200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-22T05:44:46.886 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:44:46 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[61309]: 2024-03-22T05:44:46.545+0000 7f0e20278200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-22T05:44:46.886 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:44:46 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[61309]: 2024-03-22T05:44:46.638+0000 7f0e20278200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-22T05:44:46.886 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:44:46 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[61309]: 2024-03-22T05:44:46.781+0000 7f0e20278200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-22T05:44:46.886 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:44:46 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[61309]: 2024-03-22T05:44:46.886+0000 7f0e20278200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-22T05:44:46.939 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:46 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:44:46.598+0000 7f7f883ce200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-22T05:44:46.940 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:46 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:44:46.694+0000 7f7f883ce200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-22T05:44:46.940 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:46 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:44:46.840+0000 7f7f883ce200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-22T05:44:46.940 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:46 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:44:46.939+0000 7f7f883ce200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-22T05:44:47.202 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:47 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:44:47.025+0000 7f7f883ce200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-22T05:44:47.232 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:44:46 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[61309]: 2024-03-22T05:44:46.971+0000 7f0e20278200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-22T05:44:47.505 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:47 smithi005 ceph-mon[84705]: mgrmap e45: y(active, since 3m), standbys: x 2024-03-22T05:44:47.507 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:47 smithi005 ceph-mon[29523]: mgrmap e45: y(active, since 3m), standbys: x 2024-03-22T05:44:47.507 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:47 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:44:47.300+0000 7f7f883ce200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-22T05:44:47.507 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:47 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:44:47.414+0000 7f7f883ce200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-22T05:44:47.507 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:47 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:44:47.504+0000 7f7f883ce200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-22T05:44:47.532 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:47 smithi186 ceph-mon[67903]: mgrmap e45: y(active, since 3m), standbys: x 2024-03-22T05:44:47.533 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:44:47 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[61309]: 2024-03-22T05:44:47.239+0000 7f0e20278200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-22T05:44:47.533 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:44:47 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[61309]: 2024-03-22T05:44:47.350+0000 7f0e20278200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-22T05:44:47.533 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:44:47 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[61309]: 2024-03-22T05:44:47.437+0000 7f0e20278200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-22T05:44:47.533 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:44:47 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[61309]: 2024-03-22T05:44:47.532+0000 7f0e20278200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-22T05:44:47.785 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:47 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:44:47.603+0000 7f7f883ce200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-22T05:44:47.785 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:47 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:44:47.784+0000 7f7f883ce200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-22T05:44:47.793 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:44:47 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[61309]: 2024-03-22T05:44:47.710+0000 7f0e20278200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-22T05:44:49.043 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:44:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[61309]: 2024-03-22T05:44:48.681+0000 7f0e20278200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-22T05:44:49.044 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:44:48 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[61309]: 2024-03-22T05:44:48.813+0000 7f0e20278200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-22T05:44:49.202 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:44:48.794+0000 7f7f883ce200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-22T05:44:49.202 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:44:48.931+0000 7f7f883ce200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-22T05:44:49.443 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:44:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[61309]: 2024-03-22T05:44:49.150+0000 7f0e20278200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-22T05:44:49.443 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:44:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[61309]: 2024-03-22T05:44:49.242+0000 7f0e20278200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-22T05:44:49.443 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:44:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[61309]: 2024-03-22T05:44:49.442+0000 7f0e20278200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-22T05:44:49.578 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:49 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:44:49.277+0000 7f7f883ce200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-22T05:44:49.578 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:49 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:44:49.372+0000 7f7f883ce200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-22T05:44:49.578 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:49 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:44:49.577+0000 7f7f883ce200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-22T05:44:49.793 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:44:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[61309]: 2024-03-22T05:44:49.632+0000 7f0e20278200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-22T05:44:49.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:49 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:44:49.774+0000 7f7f883ce200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-22T05:44:50.193 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:44:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[61309]: 2024-03-22T05:44:49.929+0000 7f0e20278200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-22T05:44:50.342 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:50 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:44:50.075+0000 7f7f883ce200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-22T05:44:50.543 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:44:50 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[61309]: 2024-03-22T05:44:50.192+0000 7f0e20278200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-22T05:44:50.543 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:44:50 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[61309]: 2024-03-22T05:44:50.439+0000 7f0e20278200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-22T05:44:50.597 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:50 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:44:50.342+0000 7f7f883ce200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-22T05:44:50.952 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:50 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:44:50.596+0000 7f7f883ce200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-22T05:44:51.543 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:44:51 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[61309]: 2024-03-22T05:44:51.260+0000 7f0e20278200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-22T05:44:51.951 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:44:51.453+0000 7f7f883ce200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-22T05:44:52.166 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:44:51 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[61309]: 2024-03-22T05:44:51.841+0000 7f0e20278200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-22T05:44:52.166 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:44:51 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[61309]: 2024-03-22T05:44:51.942+0000 7f0e20278200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-22T05:44:52.392 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:52 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:44:52.049+0000 7f7f883ce200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-22T05:44:52.392 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:52 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:44:52.155+0000 7f7f883ce200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-22T05:44:52.392 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:52 smithi005 ceph-mon[84705]: Standby manager daemon x restarted 2024-03-22T05:44:52.392 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:52 smithi005 ceph-mon[84705]: Standby manager daemon x started 2024-03-22T05:44:52.392 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:52 smithi005 ceph-mon[84705]: from='mgr.? 172.21.15.186:0/764277058' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-22T05:44:52.393 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:52 smithi005 ceph-mon[84705]: from='mgr.? 172.21.15.186:0/764277058' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-22T05:44:52.393 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:52 smithi005 ceph-mon[84705]: from='mgr.? 172.21.15.186:0/764277058' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-22T05:44:52.393 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:52 smithi005 ceph-mon[84705]: from='mgr.? 172.21.15.186:0/764277058' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-22T05:44:52.393 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:52 smithi005 ceph-mon[29523]: Standby manager daemon x restarted 2024-03-22T05:44:52.393 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:52 smithi005 ceph-mon[29523]: Standby manager daemon x started 2024-03-22T05:44:52.393 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:52 smithi005 ceph-mon[29523]: from='mgr.? 172.21.15.186:0/764277058' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-22T05:44:52.393 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:52 smithi005 ceph-mon[29523]: from='mgr.? 172.21.15.186:0/764277058' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-22T05:44:52.393 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:52 smithi005 ceph-mon[29523]: from='mgr.? 172.21.15.186:0/764277058' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-22T05:44:52.393 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:52 smithi005 ceph-mon[29523]: from='mgr.? 172.21.15.186:0/764277058' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-22T05:44:52.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:52 smithi186 ceph-mon[67903]: Standby manager daemon x restarted 2024-03-22T05:44:52.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:52 smithi186 ceph-mon[67903]: Standby manager daemon x started 2024-03-22T05:44:52.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:52 smithi186 ceph-mon[67903]: from='mgr.? 172.21.15.186:0/764277058' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-22T05:44:52.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:52 smithi186 ceph-mon[67903]: from='mgr.? 172.21.15.186:0/764277058' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-22T05:44:52.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:52 smithi186 ceph-mon[67903]: from='mgr.? 172.21.15.186:0/764277058' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-22T05:44:52.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:52 smithi186 ceph-mon[67903]: from='mgr.? 172.21.15.186:0/764277058' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-22T05:44:52.544 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:44:52 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[61309]: 2024-03-22T05:44:52.165+0000 7f0e20278200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-22T05:44:52.544 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:44:52 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[61309]: [22/Mar/2024:05:44:52] ENGINE Bus STARTING 2024-03-22T05:44:52.544 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:44:52 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[61309]: CherryPy Checker: 2024-03-22T05:44:52.544 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:44:52 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[61309]: The Application mounted at '' has an empty config. 2024-03-22T05:44:52.544 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:44:52 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[61309]: 2024-03-22T05:44:52.544 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:44:52 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[61309]: [22/Mar/2024:05:44:52] ENGINE Serving on http://:::9283 2024-03-22T05:44:52.544 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:44:52 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x[61309]: [22/Mar/2024:05:44:52] ENGINE Bus STARTED 2024-03-22T05:44:52.642 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:52 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:44:52.391+0000 7f7f883ce200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-22T05:44:52.643 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:52 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: [22/Mar/2024:05:44:52] ENGINE Bus STARTING 2024-03-22T05:44:52.643 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:52 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: CherryPy Checker: 2024-03-22T05:44:52.643 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:52 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: The Application mounted at '' has an empty config. 2024-03-22T05:44:52.643 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:52 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:44:52.643 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:52 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: [22/Mar/2024:05:44:52] ENGINE Serving on http://:::9283 2024-03-22T05:44:52.643 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:52 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: [22/Mar/2024:05:44:52] ENGINE Bus STARTED 2024-03-22T05:44:53.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:53 smithi186 ceph-mon[67903]: mgrmap e46: y(active, since 3m), standbys: x 2024-03-22T05:44:53.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:53 smithi186 ceph-mon[67903]: Active manager daemon y restarted 2024-03-22T05:44:53.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:53 smithi186 ceph-mon[67903]: Activating manager daemon y 2024-03-22T05:44:53.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:53 smithi186 ceph-mon[67903]: osdmap e105: 8 total, 8 up, 8 in 2024-03-22T05:44:53.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:53 smithi186 ceph-mon[67903]: mgrmap e47: y(active, starting, since 0.00639109s), standbys: x 2024-03-22T05:44:53.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:53 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-22T05:44:53.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:53 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T05:44:53.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:53 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T05:44:53.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:53 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-22T05:44:53.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:53 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-22T05:44:53.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:53 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T05:44:53.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:53 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T05:44:53.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:53 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T05:44:53.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:53 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T05:44:53.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:53 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T05:44:53.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:53 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T05:44:53.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:53 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-22T05:44:53.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:53 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-22T05:44:53.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:53 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-22T05:44:53.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:53 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-22T05:44:53.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:53 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-22T05:44:53.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:53 smithi186 ceph-mon[67903]: Manager daemon y is now available 2024-03-22T05:44:53.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:53 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:44:53.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:53 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:44:53.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:53 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-22T05:44:53.546 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:53 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-22T05:44:53.546 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:53 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-22T05:44:53.546 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:53 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-22T05:44:53.638 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:53 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:44:53.423+0000 7f7f3e571700 -1 mgr.server handle_report got status from non-daemon mon.a 2024-03-22T05:44:53.638 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:53 smithi005 ceph-mon[84705]: mgrmap e46: y(active, since 3m), standbys: x 2024-03-22T05:44:53.638 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:53 smithi005 ceph-mon[84705]: Active manager daemon y restarted 2024-03-22T05:44:53.638 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:53 smithi005 ceph-mon[84705]: Activating manager daemon y 2024-03-22T05:44:53.638 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:53 smithi005 ceph-mon[84705]: osdmap e105: 8 total, 8 up, 8 in 2024-03-22T05:44:53.639 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:53 smithi005 ceph-mon[84705]: mgrmap e47: y(active, starting, since 0.00639109s), standbys: x 2024-03-22T05:44:53.639 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:53 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-22T05:44:53.639 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:53 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T05:44:53.639 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:53 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T05:44:53.639 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:53 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-22T05:44:53.639 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:53 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-22T05:44:53.639 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:53 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T05:44:53.639 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:53 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T05:44:53.639 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:53 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T05:44:53.639 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:53 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T05:44:53.639 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:53 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T05:44:53.639 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:53 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T05:44:53.639 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:53 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-22T05:44:53.639 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:53 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-22T05:44:53.639 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:53 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-22T05:44:53.640 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:53 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-22T05:44:53.640 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:53 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-22T05:44:53.640 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:53 smithi005 ceph-mon[84705]: Manager daemon y is now available 2024-03-22T05:44:53.640 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:53 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:44:53.640 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:53 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:44:53.640 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:53 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-22T05:44:53.640 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:53 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-22T05:44:53.640 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:53 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-22T05:44:53.640 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:53 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-22T05:44:53.640 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:53 smithi005 ceph-mon[29523]: mgrmap e46: y(active, since 3m), standbys: x 2024-03-22T05:44:53.640 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:53 smithi005 ceph-mon[29523]: Active manager daemon y restarted 2024-03-22T05:44:53.640 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:53 smithi005 ceph-mon[29523]: Activating manager daemon y 2024-03-22T05:44:53.641 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:53 smithi005 ceph-mon[29523]: osdmap e105: 8 total, 8 up, 8 in 2024-03-22T05:44:53.641 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:53 smithi005 ceph-mon[29523]: mgrmap e47: y(active, starting, since 0.00639109s), standbys: x 2024-03-22T05:44:53.641 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:53 smithi005 ceph-mon[29523]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-22T05:44:53.641 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:53 smithi005 ceph-mon[29523]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T05:44:53.641 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:53 smithi005 ceph-mon[29523]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T05:44:53.641 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:53 smithi005 ceph-mon[29523]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-22T05:44:53.641 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:53 smithi005 ceph-mon[29523]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-22T05:44:53.641 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:53 smithi005 ceph-mon[29523]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T05:44:53.641 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:53 smithi005 ceph-mon[29523]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T05:44:53.641 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:53 smithi005 ceph-mon[29523]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T05:44:53.641 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:53 smithi005 ceph-mon[29523]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T05:44:53.641 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:53 smithi005 ceph-mon[29523]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T05:44:53.641 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:53 smithi005 ceph-mon[29523]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T05:44:53.641 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:53 smithi005 ceph-mon[29523]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-22T05:44:53.641 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:53 smithi005 ceph-mon[29523]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-22T05:44:53.642 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:53 smithi005 ceph-mon[29523]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-22T05:44:53.642 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:53 smithi005 ceph-mon[29523]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-22T05:44:53.642 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:53 smithi005 ceph-mon[29523]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-22T05:44:53.642 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:53 smithi005 ceph-mon[29523]: Manager daemon y is now available 2024-03-22T05:44:53.642 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:53 smithi005 ceph-mon[29523]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:44:53.642 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:53 smithi005 ceph-mon[29523]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:44:53.642 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:53 smithi005 ceph-mon[29523]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-22T05:44:53.642 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:53 smithi005 ceph-mon[29523]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-22T05:44:53.642 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:53 smithi005 ceph-mon[29523]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-22T05:44:53.642 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:53 smithi005 ceph-mon[29523]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-22T05:44:54.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:54 smithi005 ceph-mon[84705]: mgrmap e48: y(active, since 1.01232s), standbys: x 2024-03-22T05:44:54.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:54 smithi005 ceph-mon[84705]: [22/Mar/2024:05:44:54] ENGINE Bus STARTING 2024-03-22T05:44:54.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:54 smithi005 ceph-mon[84705]: [22/Mar/2024:05:44:54] ENGINE Serving on https://172.21.15.5:7150 2024-03-22T05:44:54.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:54 smithi005 ceph-mon[29523]: mgrmap e48: y(active, since 1.01232s), standbys: x 2024-03-22T05:44:54.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:54 smithi005 ceph-mon[29523]: [22/Mar/2024:05:44:54] ENGINE Bus STARTING 2024-03-22T05:44:54.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:54 smithi005 ceph-mon[29523]: [22/Mar/2024:05:44:54] ENGINE Serving on https://172.21.15.5:7150 2024-03-22T05:44:54.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:54 smithi186 ceph-mon[67903]: mgrmap e48: y(active, since 1.01232s), standbys: x 2024-03-22T05:44:54.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:54 smithi186 ceph-mon[67903]: [22/Mar/2024:05:44:54] ENGINE Bus STARTING 2024-03-22T05:44:54.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:54 smithi186 ceph-mon[67903]: [22/Mar/2024:05:44:54] ENGINE Serving on https://172.21.15.5:7150 2024-03-22T05:44:55.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:55 smithi005 ceph-mon[29523]: [22/Mar/2024:05:44:54] ENGINE Serving on http://172.21.15.5:8765 2024-03-22T05:44:55.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:55 smithi005 ceph-mon[29523]: [22/Mar/2024:05:44:54] ENGINE Bus STARTED 2024-03-22T05:44:55.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:55 smithi005 ceph-mon[29523]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-22T05:44:55.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:55 smithi005 ceph-mon[29523]: Cluster is now healthy 2024-03-22T05:44:55.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:55 smithi005 ceph-mon[29523]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:44:55.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:55 smithi005 ceph-mon[84705]: [22/Mar/2024:05:44:54] ENGINE Serving on http://172.21.15.5:8765 2024-03-22T05:44:55.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:55 smithi005 ceph-mon[84705]: [22/Mar/2024:05:44:54] ENGINE Bus STARTED 2024-03-22T05:44:55.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:55 smithi005 ceph-mon[84705]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-22T05:44:55.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:55 smithi005 ceph-mon[84705]: Cluster is now healthy 2024-03-22T05:44:55.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:55 smithi005 ceph-mon[84705]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:44:55.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:55 smithi186 ceph-mon[67903]: [22/Mar/2024:05:44:54] ENGINE Serving on http://172.21.15.5:8765 2024-03-22T05:44:55.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:55 smithi186 ceph-mon[67903]: [22/Mar/2024:05:44:54] ENGINE Bus STARTED 2024-03-22T05:44:55.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:55 smithi186 ceph-mon[67903]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-22T05:44:55.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:55 smithi186 ceph-mon[67903]: Cluster is now healthy 2024-03-22T05:44:55.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:55 smithi186 ceph-mon[67903]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:44:56.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:56 smithi005 ceph-mon[84705]: mgrmap e49: y(active, since 3s), standbys: x 2024-03-22T05:44:56.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:56 smithi005 ceph-mon[29523]: mgrmap e49: y(active, since 3s), standbys: x 2024-03-22T05:44:56.757 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:56 smithi186 ceph-mon[67903]: mgrmap e49: y(active, since 3s), standbys: x 2024-03-22T05:44:57.202 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:44:56 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:44:56] "GET /metrics HTTP/1.1" 200 35196 "" "Prometheus/2.43.0" 2024-03-22T05:44:58.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:58 smithi005 ceph-mon[29523]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:44:58.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:58 smithi005 ceph-mon[29523]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:44:58.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:58 smithi005 ceph-mon[29523]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:44:58.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:58 smithi005 ceph-mon[29523]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:44:58.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:58 smithi005 ceph-mon[29523]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:44:58.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:58 smithi005 ceph-mon[29523]: mgrmap e50: y(active, since 5s), standbys: x 2024-03-22T05:44:58.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:58 smithi005 ceph-mon[84705]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:44:58.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:58 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:44:58.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:58 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:44:58.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:58 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:44:58.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:58 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:44:58.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:58 smithi005 ceph-mon[84705]: mgrmap e50: y(active, since 5s), standbys: x 2024-03-22T05:44:58.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:58 smithi186 ceph-mon[67903]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:44:58.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:58 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:44:58.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:58 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:44:58.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:58 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:44:58.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:58 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:44:58.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:58 smithi186 ceph-mon[67903]: mgrmap e50: y(active, since 5s), standbys: x 2024-03-22T05:44:59.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:59 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:44:59.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:59 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:44:59.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:59 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:44:59.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:44:59 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:44:59.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:59 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:44:59.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:59 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:44:59.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:59 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:44:59.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:44:59 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:44:59.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:59 smithi005 ceph-mon[29523]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:44:59.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:59 smithi005 ceph-mon[29523]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:44:59.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:59 smithi005 ceph-mon[29523]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:44:59.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:44:59 smithi005 ceph-mon[29523]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:00.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:00 smithi005 ceph-mon[84705]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:45:00.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:00 smithi005 ceph-mon[29523]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:45:00.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:00 smithi186 ceph-mon[67903]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:45:02.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:02 smithi186 ceph-mon[67903]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-22T05:45:02.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:02 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:02.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:02 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:02.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:02 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:02.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:02 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:02.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:02 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:45:02.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:02 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:45:02.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:02 smithi005 ceph-mon[84705]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-22T05:45:02.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:02 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:02.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:02 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:02.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:02 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:02.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:02 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:02.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:02 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:45:02.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:02 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:45:02.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:02 smithi005 ceph-mon[29523]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-22T05:45:02.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:02 smithi005 ceph-mon[29523]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:02.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:02 smithi005 ceph-mon[29523]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:02.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:02 smithi005 ceph-mon[29523]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:02.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:02 smithi005 ceph-mon[29523]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:02.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:02 smithi005 ceph-mon[29523]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:45:02.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:02 smithi005 ceph-mon[29523]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:45:03.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:03 smithi186 ceph-mon[67903]: Detected new or changed devices on smithi005 2024-03-22T05:45:03.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:03 smithi186 ceph-mon[67903]: Updating smithi005:/etc/ceph/ceph.conf 2024-03-22T05:45:03.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:03 smithi186 ceph-mon[67903]: Updating smithi186:/etc/ceph/ceph.conf 2024-03-22T05:45:03.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:03 smithi186 ceph-mon[67903]: Updating smithi186:/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T05:45:03.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:03 smithi186 ceph-mon[67903]: Updating smithi005:/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T05:45:03.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:03 smithi186 ceph-mon[67903]: Updating smithi186:/etc/ceph/ceph.client.admin.keyring 2024-03-22T05:45:03.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:03 smithi186 ceph-mon[67903]: Updating smithi005:/etc/ceph/ceph.client.admin.keyring 2024-03-22T05:45:03.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:03 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:03.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:03 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:03.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:03 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:03.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:03 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:03.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:03 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:03.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:03 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:45:03.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:03 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:45:03.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:03 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:45:03.611 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:03 smithi005 ceph-mon[29523]: Detected new or changed devices on smithi005 2024-03-22T05:45:03.611 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:03 smithi005 ceph-mon[29523]: Updating smithi005:/etc/ceph/ceph.conf 2024-03-22T05:45:03.611 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:03 smithi005 ceph-mon[29523]: Updating smithi186:/etc/ceph/ceph.conf 2024-03-22T05:45:03.611 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:03 smithi005 ceph-mon[29523]: Updating smithi186:/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T05:45:03.611 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:03 smithi005 ceph-mon[29523]: Updating smithi005:/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T05:45:03.611 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:03 smithi005 ceph-mon[29523]: Updating smithi186:/etc/ceph/ceph.client.admin.keyring 2024-03-22T05:45:03.611 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:03 smithi005 ceph-mon[29523]: Updating smithi005:/etc/ceph/ceph.client.admin.keyring 2024-03-22T05:45:03.611 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:03 smithi005 ceph-mon[29523]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:03.611 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:03 smithi005 ceph-mon[29523]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:03.611 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:03 smithi005 ceph-mon[29523]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:03.611 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:03 smithi005 ceph-mon[29523]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:03.611 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:03 smithi005 ceph-mon[29523]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:03.611 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:03 smithi005 ceph-mon[29523]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:45:03.612 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:03 smithi005 ceph-mon[29523]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:45:03.612 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:03 smithi005 ceph-mon[29523]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:45:03.612 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:03 smithi005 ceph-mon[84705]: Detected new or changed devices on smithi005 2024-03-22T05:45:03.612 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:03 smithi005 ceph-mon[84705]: Updating smithi005:/etc/ceph/ceph.conf 2024-03-22T05:45:03.612 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:03 smithi005 ceph-mon[84705]: Updating smithi186:/etc/ceph/ceph.conf 2024-03-22T05:45:03.612 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:03 smithi005 ceph-mon[84705]: Updating smithi186:/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T05:45:03.612 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:03 smithi005 ceph-mon[84705]: Updating smithi005:/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-22T05:45:03.612 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:03 smithi005 ceph-mon[84705]: Updating smithi186:/etc/ceph/ceph.client.admin.keyring 2024-03-22T05:45:03.612 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:03 smithi005 ceph-mon[84705]: Updating smithi005:/etc/ceph/ceph.client.admin.keyring 2024-03-22T05:45:03.612 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:03 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:03.612 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:03 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:03.613 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:03 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:03.613 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:03 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:03.613 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:03 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:03.613 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:03 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:45:03.613 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:03 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:45:03.613 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:03 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:45:04.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:04 smithi186 ceph-mon[67903]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-22T05:45:04.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:04 smithi186 ceph-mon[67903]: Updating smithi186:/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-22T05:45:04.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:04 smithi186 ceph-mon[67903]: Updating smithi005:/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-22T05:45:04.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:04 smithi186 ceph-mon[67903]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-22T05:45:04.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:04 smithi186 ceph-mon[67903]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:45:04.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:04 smithi186 ceph-mon[67903]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:45:04.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:04 smithi186 ceph-mon[67903]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-22T05:45:04.693 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:04 smithi005 ceph-mon[84705]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-22T05:45:04.693 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:04 smithi005 ceph-mon[84705]: Updating smithi186:/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-22T05:45:04.693 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:04 smithi005 ceph-mon[84705]: Updating smithi005:/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-22T05:45:04.693 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:04 smithi005 ceph-mon[84705]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-22T05:45:04.693 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:04 smithi005 ceph-mon[84705]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:45:04.693 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:04 smithi005 ceph-mon[84705]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:45:04.693 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:04 smithi005 ceph-mon[84705]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-22T05:45:04.694 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:04 smithi005 ceph-mon[29523]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-22T05:45:04.694 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:04 smithi005 ceph-mon[29523]: Updating smithi186:/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-22T05:45:04.694 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:04 smithi005 ceph-mon[29523]: Updating smithi005:/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-22T05:45:04.694 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:04 smithi005 ceph-mon[29523]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-22T05:45:04.694 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:04 smithi005 ceph-mon[29523]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:45:04.694 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:04 smithi005 ceph-mon[29523]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:45:04.694 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:04 smithi005 ceph-mon[29523]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-22T05:45:04.952 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:45:04.852+0000 7f7f0ecc9700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:45:04.952 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:45:04.952 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:45:04.952 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:45:04.952 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:45:04.952 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:45:04.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:45:04.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:45:04.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:45:04.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:45:04.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:45:04.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:45:04.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:45:04.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:45:04.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:45:04.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:45:04.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:45:04.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:45:04.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:45:04.954 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:45:04.954 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:45:04.954 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:45:04.954 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:45:04.954 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:45:04.954 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:45:04.954 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:45:04.954 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:45:04.954 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:45:04.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:45:04.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return future.result(timeout) 2024-03-22T05:45:04.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:45:04.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.__get_result() 2024-03-22T05:45:04.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:45:04.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: raise self._exception 2024-03-22T05:45:04.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:45:04.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: config_blobs=daemon_spec.final_config, 2024-03-22T05:45:04.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:45:04.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:45:04.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:45:04.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:45:04.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:45:04.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:45:04.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:45:04.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:45:04.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:45:04.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:45:04.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:45:04.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:45:04.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:45:04.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:45:04.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:45:04.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:45:04.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:45:04.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:45:04.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:45:04.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:45:04.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:45:04.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:45:04.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:45:05.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:05 smithi186 ceph-mon[67903]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:45:05.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:05 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:45:05.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:05 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:45:05.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:05 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-22T05:45:05.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:05 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-03-22T05:45:05.572 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:05 smithi005 ceph-mon[29523]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:45:05.572 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:05 smithi005 ceph-mon[29523]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:45:05.572 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:05 smithi005 ceph-mon[29523]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:45:05.572 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:05 smithi005 ceph-mon[29523]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-22T05:45:05.572 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:05 smithi005 ceph-mon[29523]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-03-22T05:45:05.572 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:05 smithi005 ceph-mon[84705]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:45:05.572 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:05 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:45:05.572 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:05 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:45:05.572 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:05 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-22T05:45:05.572 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:05 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-03-22T05:45:06.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:06 smithi186 ceph-mon[67903]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-22T05:45:06.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:06 smithi186 ceph-mon[67903]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:45:06.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:45:06.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:45:06.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:45:06.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:45:06.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:45:06.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:45:06.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:45:06.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:45:06.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:45:06.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:45:06.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:45:06.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:45:06.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:45:06.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:45:06.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:45:06.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:45:06.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:45:06.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:45:06.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:45:06.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:45:06.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:45:06.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:45:06.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:45:06.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:45:06.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:45:06.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:45:06.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:45:06.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:45:06.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: return future.result(timeout) 2024-03-22T05:45:06.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:45:06.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.__get_result() 2024-03-22T05:45:06.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:45:06.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: raise self._exception 2024-03-22T05:45:06.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:45:06.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:45:06.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:45:06.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:45:06.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:45:06.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:45:06.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:45:06.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:45:06.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:45:06.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:45:06.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:45:06.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:45:06.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:45:06.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:45:06.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:45:06.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:45:06.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:45:06.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:45:06.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:45:06.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:45:06.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:45:06.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:45:06.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:45:06.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:45:06.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:45:06.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:06 smithi186 ceph-mon[67903]: Upgrade: It appears safe to stop mon.c 2024-03-22T05:45:06.576 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:06 smithi005 ceph-mon[29523]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-22T05:45:06.576 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:06 smithi005 ceph-mon[29523]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:45:06.576 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:45:06.577 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:45:06.577 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:45:06.577 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:45:06.577 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:45:06.577 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:45:06.577 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:45:06.577 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:45:06.577 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:45:06.577 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:45:06.577 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:45:06.577 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:45:06.577 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:45:06.577 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:45:06.577 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:45:06.578 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:45:06.578 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:45:06.578 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:45:06.578 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:45:06.578 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:45:06.578 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:45:06.578 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:45:06.578 INFO:journalctl@ceph.mon.c.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:45:06.578 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:45:06.578 INFO:journalctl@ceph.mon.c.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:45:06.578 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:45:06.578 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:45:06.578 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:45:06.578 INFO:journalctl@ceph.mon.c.smithi005.stdout: return future.result(timeout) 2024-03-22T05:45:06.579 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:45:06.579 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.__get_result() 2024-03-22T05:45:06.579 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:45:06.579 INFO:journalctl@ceph.mon.c.smithi005.stdout: raise self._exception 2024-03-22T05:45:06.579 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:45:06.579 INFO:journalctl@ceph.mon.c.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:45:06.579 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:45:06.579 INFO:journalctl@ceph.mon.c.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:45:06.579 INFO:journalctl@ceph.mon.c.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:45:06.579 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:45:06.579 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:45:06.579 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:45:06.579 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:45:06.579 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:45:06.580 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:45:06.580 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:45:06.580 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:45:06.580 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:45:06.580 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:45:06.580 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:45:06.580 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:45:06.580 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:45:06.580 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:45:06.580 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:45:06.580 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:45:06.580 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:45:06.580 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:45:06.580 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:45:06.581 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:45:06.581 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:06 smithi005 ceph-mon[29523]: Upgrade: It appears safe to stop mon.c 2024-03-22T05:45:06.581 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:06 smithi005 ceph-mon[84705]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-22T05:45:06.581 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:06 smithi005 ceph-mon[84705]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:45:06.581 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:45:06.581 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:45:06.581 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:45:06.581 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:45:06.581 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:45:06.582 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:45:06.582 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:45:06.582 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:45:06.582 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:45:06.582 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:45:06.582 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:45:06.582 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:45:06.582 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:45:06.582 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:45:06.582 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:45:06.582 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:45:06.582 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:45:06.582 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:45:06.582 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:45:06.582 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:45:06.583 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:45:06.583 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:45:06.583 INFO:journalctl@ceph.mon.a.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:45:06.583 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:45:06.583 INFO:journalctl@ceph.mon.a.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:45:06.583 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:45:06.583 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:45:06.583 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:45:06.583 INFO:journalctl@ceph.mon.a.smithi005.stdout: return future.result(timeout) 2024-03-22T05:45:06.583 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:45:06.583 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.__get_result() 2024-03-22T05:45:06.583 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:45:06.583 INFO:journalctl@ceph.mon.a.smithi005.stdout: raise self._exception 2024-03-22T05:45:06.583 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:45:06.583 INFO:journalctl@ceph.mon.a.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:45:06.584 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:45:06.584 INFO:journalctl@ceph.mon.a.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:45:06.584 INFO:journalctl@ceph.mon.a.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:45:06.584 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:45:06.584 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:45:06.584 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:45:06.584 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:45:06.584 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:45:06.584 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:45:06.584 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:45:06.584 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:45:06.584 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:45:06.584 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:45:06.584 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:45:06.584 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:45:06.585 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:45:06.585 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:45:06.585 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:45:06.585 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:45:06.585 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:45:06.585 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:45:06.585 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:45:06.585 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:45:06.585 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:06 smithi005 ceph-mon[84705]: Upgrade: It appears safe to stop mon.c 2024-03-22T05:45:07.211 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:45:06] "GET /metrics HTTP/1.1" 200 35196 "" "Prometheus/2.43.0" 2024-03-22T05:45:07.410 INFO:teuthology.orchestra.run.smithi005.stdout:true 2024-03-22T05:45:07.689 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:07 smithi005 ceph-mon[84705]: Upgrade: Updating mon.c 2024-03-22T05:45:07.689 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:07 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:07.689 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:07 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-22T05:45:07.689 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:07 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-22T05:45:07.690 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:07 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:45:07.690 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:07 smithi005 ceph-mon[84705]: Deploying daemon mon.c on smithi005 2024-03-22T05:45:07.690 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:07 smithi005 ceph-mon[84705]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-22T05:45:07.690 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:07 smithi005 ceph-mon[29523]: Upgrade: Updating mon.c 2024-03-22T05:45:07.690 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:07 smithi005 ceph-mon[29523]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:07.690 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:07 smithi005 ceph-mon[29523]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-22T05:45:07.690 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:07 smithi005 ceph-mon[29523]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-22T05:45:07.690 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:07 smithi005 ceph-mon[29523]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:45:07.690 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:07 smithi005 ceph-mon[29523]: Deploying daemon mon.c on smithi005 2024-03-22T05:45:07.690 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:07 smithi005 ceph-mon[29523]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-22T05:45:07.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:07 smithi186 ceph-mon[67903]: Upgrade: Updating mon.c 2024-03-22T05:45:07.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:07 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:07.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:07 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-22T05:45:07.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:07 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-22T05:45:07.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:07 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:45:07.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:07 smithi186 ceph-mon[67903]: Deploying daemon mon.c on smithi005 2024-03-22T05:45:07.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:07 smithi186 ceph-mon[67903]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-22T05:45:08.095 INFO:teuthology.orchestra.run.smithi005.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T05:45:08.095 INFO:teuthology.orchestra.run.smithi005.stdout:alertmanager.a smithi005 *:9093,9094 running (2m) 10s ago 24m 19.7M - 0.25.0 c8568f914cd2 ee99dfc435e8 2024-03-22T05:45:08.096 INFO:teuthology.orchestra.run.smithi005.stdout:grafana.a smithi186 *:3000 running (2m) 10s ago 24m 81.5M - 9.4.12 2bacad6d85d8 16fd57dbbd71 2024-03-22T05:45:08.096 INFO:teuthology.orchestra.run.smithi005.stdout:iscsi.foo.smithi005.mcdwbp smithi005 unknown 10s ago 23m 61.4M - 3.5 e1d6a67b021e 5215e03bf81b 2024-03-22T05:45:08.096 INFO:teuthology.orchestra.run.smithi005.stdout:mgr.x smithi186 *:8443,9283,8765 running (3m) 10s ago 29m 448M - 19.0.0-2299-gea01217f aa461e7c5e7f c556421a73f3 2024-03-22T05:45:08.096 INFO:teuthology.orchestra.run.smithi005.stdout:mgr.y smithi005 *:8443,9283,8765 running (13m) 10s ago 30m 473M - 19.0.0-2299-gea01217f aa461e7c5e7f 62a813eebc0d 2024-03-22T05:45:08.096 INFO:teuthology.orchestra.run.smithi005.stdout:mon.a smithi005 running (26s) 10s ago 30m 34.4M 2048M 19.0.0-2299-gea01217f aa461e7c5e7f 111a2aa24dd9 2024-03-22T05:45:08.096 INFO:teuthology.orchestra.run.smithi005.stdout:mon.b smithi186 running (76s) 10s ago 29m 39.1M 2048M 19.0.0-2299-gea01217f aa461e7c5e7f c1798feac15b 2024-03-22T05:45:08.096 INFO:teuthology.orchestra.run.smithi005.stdout:mon.c smithi005 running (29m) 10s ago 29m 64.6M 2048M 17.2.0 e1d6a67b021e 00ec24d38612 2024-03-22T05:45:08.096 INFO:teuthology.orchestra.run.smithi005.stdout:node-exporter.a smithi005 *:9100 running (3m) 10s ago 25m 8400k - 1.5.0 0da6a335fe13 f635cd72b9d3 2024-03-22T05:45:08.096 INFO:teuthology.orchestra.run.smithi005.stdout:node-exporter.b smithi186 *:9100 running (3m) 10s ago 25m 15.1M - 1.5.0 0da6a335fe13 d7e0b3d20191 2024-03-22T05:45:08.096 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0 smithi005 running (28m) 10s ago 28m 53.7M 2993M 17.2.0 e1d6a67b021e ef5c7053dd37 2024-03-22T05:45:08.096 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1 smithi005 running (28m) 10s ago 28m 56.6M 2993M 17.2.0 e1d6a67b021e a1e7ee1017b6 2024-03-22T05:45:08.096 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2 smithi005 running (27m) 10s ago 27m 48.7M 2993M 17.2.0 e1d6a67b021e 462ea63ad0e5 2024-03-22T05:45:08.096 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3 smithi005 running (27m) 10s ago 27m 56.6M 2993M 17.2.0 e1d6a67b021e b25432a7239b 2024-03-22T05:45:08.096 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4 smithi186 running (27m) 10s ago 27m 56.4M 3248M 17.2.0 e1d6a67b021e cd963671f340 2024-03-22T05:45:08.097 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5 smithi186 running (26m) 10s ago 26m 51.7M 3248M 17.2.0 e1d6a67b021e f98c87f1a2cd 2024-03-22T05:45:08.097 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6 smithi186 running (26m) 10s ago 26m 52.0M 3248M 17.2.0 e1d6a67b021e 704ac49c68c4 2024-03-22T05:45:08.097 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7 smithi186 running (26m) 10s ago 26m 54.3M 3248M 17.2.0 e1d6a67b021e f415b90edf63 2024-03-22T05:45:08.097 INFO:teuthology.orchestra.run.smithi005.stdout:prometheus.a smithi186 *:9095 running (2m) 10s ago 25m 67.2M - 2.43.0 a07b618ecd1d a5a043588d6d 2024-03-22T05:45:08.097 INFO:teuthology.orchestra.run.smithi005.stdout:rgw.foo.smithi005.exdhqu smithi005 *:8000 running (23m) 10s ago 23m 79.9M - 17.2.0 e1d6a67b021e 6bb27fe11553 2024-03-22T05:45:08.097 INFO:teuthology.orchestra.run.smithi005.stdout:rgw.foo.smithi186.mcgodk smithi186 *:8000 running (23m) 10s ago 23m 78.2M - 17.2.0 e1d6a67b021e 19b5c20a34f7 2024-03-22T05:45:08.275 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:08 smithi005 systemd[1]: Stopping Ceph mon.c for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:45:08.554 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-03-22T05:45:08.554 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": { 2024-03-22T05:45:08.554 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-03-22T05:45:08.554 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev)": 2 2024-03-22T05:45:08.554 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:45:08.554 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr": { 2024-03-22T05:45:08.554 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev)": 2 2024-03-22T05:45:08.554 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:45:08.555 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": { 2024-03-22T05:45:08.555 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-22T05:45:08.555 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:45:08.555 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw": { 2024-03-22T05:45:08.555 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-22T05:45:08.555 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:45:08.555 INFO:teuthology.orchestra.run.smithi005.stdout: "overall": { 2024-03-22T05:45:08.555 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 11, 2024-03-22T05:45:08.555 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev)": 4 2024-03-22T05:45:08.555 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-03-22T05:45:08.555 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-03-22T05:45:08.555 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:08 smithi005 ceph-mon[84705]: from='client.25645 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:45:08.556 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:08 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:45:08.556 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:08 smithi005 ceph-mon[84705]: from='client.25651 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:45:08.556 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:08 smithi005 ceph-mon[84705]: from='client.54150 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:45:08.556 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:08 smithi005 ceph-mon[29523]: from='client.25645 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:45:08.556 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:08 smithi005 ceph-mon[29523]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:45:08.556 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:08 smithi005 ceph-mon[29523]: from='client.25651 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:45:08.556 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:08 smithi005 ceph-mon[29523]: from='client.54150 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:45:08.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:08 smithi186 ceph-mon[67903]: from='client.25645 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:45:08.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:08 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:45:08.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:08 smithi186 ceph-mon[67903]: from='client.25651 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:45:08.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:08 smithi186 ceph-mon[67903]: from='client.54150 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:45:08.904 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-03-22T05:45:08.904 INFO:teuthology.orchestra.run.smithi005.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9", 2024-03-22T05:45:08.905 INFO:teuthology.orchestra.run.smithi005.stdout: "in_progress": true, 2024-03-22T05:45:08.905 INFO:teuthology.orchestra.run.smithi005.stdout: "which": "Upgrading daemons of type(s) mon on host(s) smithi005", 2024-03-22T05:45:08.905 INFO:teuthology.orchestra.run.smithi005.stdout: "services_complete": [], 2024-03-22T05:45:08.905 INFO:teuthology.orchestra.run.smithi005.stdout: "progress": "1/2 daemons upgraded", 2024-03-22T05:45:08.905 INFO:teuthology.orchestra.run.smithi005.stdout: "message": "Currently upgrading mon daemons", 2024-03-22T05:45:08.905 INFO:teuthology.orchestra.run.smithi005.stdout: "is_paused": false 2024-03-22T05:45:08.905 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-03-22T05:45:08.905 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:08 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mon-c[29519]: 2024-03-22T05:45:08.578+0000 7f6a1d5d8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T05:45:08.905 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:08 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mon-c[29519]: 2024-03-22T05:45:08.578+0000 7f6a1d5d8700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-03-22T05:45:08.905 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:08 smithi005 podman[89714]: 2024-03-22 05:45:08.660865883 +0000 UTC m=+0.219584653 container died 00ec24d38612c9e5b2bd914999a6c666bcc53331ecb8cfd79900c97c98e16c16 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mon-c, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, architecture=x86_64, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, version=8, distribution-scope=public, GIT_CLEAN=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0) 2024-03-22T05:45:09.159 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:09 smithi005 podman[89714]: 2024-03-22 05:45:09.145315102 +0000 UTC m=+0.704033868 container cleanup 00ec24d38612c9e5b2bd914999a6c666bcc53331ecb8cfd79900c97c98e16c16 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mon-c, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, io.openshift.expose-services=, name=centos-stream, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, architecture=x86_64, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, vendor=Red Hat, Inc., io.buildah.version=1.19.8, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-03-22T05:45:09.160 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:09 smithi005 bash[89714]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mon-c 2024-03-22T05:45:09.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:09 smithi005 podman[89732]: 2024-03-22 05:45:09.413541322 +0000 UTC m=+0.756526695 container remove 00ec24d38612c9e5b2bd914999a6c666bcc53331ecb8cfd79900c97c98e16c16 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mon-c, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, architecture=x86_64, release=754, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, vcs-type=git, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-03-22T05:45:09.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:09 smithi005 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@mon.c.service: Deactivated successfully. 2024-03-22T05:45:09.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:09 smithi005 systemd[1]: Stopped Ceph mon.c for f59f36e2-e80a-11ee-95cd-87774f69a715. 2024-03-22T05:45:09.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:09 smithi005 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@mon.c.service: Consumed 16.752s CPU time. 2024-03-22T05:45:10.077 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:09 smithi005 systemd[1]: Starting Ceph mon.c for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:45:10.077 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:09 smithi005 podman[89884]: 2024-03-22 05:45:09.981286218 +0000 UTC m=+0.114960646 container create 708c72ff74f5b841a258f70d64a4f638b6da5c5ea37a4ba779c0151f63a841c5 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mon-c, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-22T05:45:10.077 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:09 smithi005 podman[89884]: 2024-03-22 05:45:09.884991733 +0000 UTC m=+0.018666132 image pull aa461e7c5e7f322b169c789b389d594b994ea7ffcbc9e24e563a3ec9b69bc3c1 quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9 2024-03-22T05:45:10.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 podman[89884]: 2024-03-22 05:45:10.183634412 +0000 UTC m=+0.317308813 container init 708c72ff74f5b841a258f70d64a4f638b6da5c5ea37a4ba779c0151f63a841c5 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mon-c, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319) 2024-03-22T05:45:10.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 podman[89884]: 2024-03-22 05:45:10.1865976 +0000 UTC m=+0.320271988 container start 708c72ff74f5b841a258f70d64a4f638b6da5c5ea37a4ba779c0151f63a841c5 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mon-c, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-22T05:45:10.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: set uid:gid to 167:167 (ceph:ceph) 2024-03-22T05:45:10.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev), process ceph-mon, pid 2 2024-03-22T05:45:10.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: pidfile_write: ignore empty --pid-file 2024-03-22T05:45:10.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: load: jerasure load: lrc 2024-03-22T05:45:10.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: RocksDB version: 7.9.2 2024-03-22T05:45:10.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Git sha 0 2024-03-22T05:45:10.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Compile date 2024-03-21 23:09:57 2024-03-22T05:45:10.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: DB SUMMARY 2024-03-22T05:45:10.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: DB Session ID: RU8ASWXNDTG7A8J3O3MJ 2024-03-22T05:45:10.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: CURRENT file: CURRENT 2024-03-22T05:45:10.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: IDENTITY file: IDENTITY 2024-03-22T05:45:10.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: MANIFEST file: MANIFEST-000009 size: 3196 Bytes 2024-03-22T05:45:10.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 1, files: 000060.sst 2024-03-22T05:45:10.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000058.log size: 2783649 ; 2024-03-22T05:45:10.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.error_if_exists: 0 2024-03-22T05:45:10.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.create_if_missing: 0 2024-03-22T05:45:10.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.paranoid_checks: 1 2024-03-22T05:45:10.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-22T05:45:10.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-22T05:45:10.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-22T05:45:10.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.env: 0x564508d09b20 2024-03-22T05:45:10.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.fs: PosixFileSystem 2024-03-22T05:45:10.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.info_log: 0x564509fb9920 2024-03-22T05:45:10.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.max_file_opening_threads: 16 2024-03-22T05:45:10.456 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.statistics: (nil) 2024-03-22T05:45:10.456 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.use_fsync: 0 2024-03-22T05:45:10.456 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.max_log_file_size: 0 2024-03-22T05:45:10.456 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-22T05:45:10.456 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-22T05:45:10.457 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.keep_log_file_num: 1000 2024-03-22T05:45:10.457 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.recycle_log_file_num: 0 2024-03-22T05:45:10.457 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.allow_fallocate: 1 2024-03-22T05:45:10.457 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.allow_mmap_reads: 0 2024-03-22T05:45:10.457 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.allow_mmap_writes: 0 2024-03-22T05:45:10.457 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.use_direct_reads: 0 2024-03-22T05:45:10.457 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-22T05:45:10.457 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.create_missing_column_families: 0 2024-03-22T05:45:10.457 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.db_log_dir: 2024-03-22T05:45:10.457 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.wal_dir: 2024-03-22T05:45:10.458 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-22T05:45:10.458 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-22T05:45:10.458 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-22T05:45:10.458 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-22T05:45:10.458 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-22T05:45:10.458 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-22T05:45:10.458 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.advise_random_on_open: 1 2024-03-22T05:45:10.458 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.db_write_buffer_size: 0 2024-03-22T05:45:10.459 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.write_buffer_manager: 0x564509f8a1e0 2024-03-22T05:45:10.459 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-22T05:45:10.459 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-22T05:45:10.459 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-22T05:45:10.459 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.rate_limiter: (nil) 2024-03-22T05:45:10.459 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-22T05:45:10.459 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.wal_recovery_mode: 2 2024-03-22T05:45:10.459 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.enable_thread_tracking: 0 2024-03-22T05:45:10.459 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.enable_pipelined_write: 0 2024-03-22T05:45:10.459 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.unordered_write: 0 2024-03-22T05:45:10.460 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-22T05:45:10.460 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-22T05:45:10.460 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-22T05:45:10.460 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-22T05:45:10.460 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.row_cache: None 2024-03-22T05:45:10.460 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.wal_filter: None 2024-03-22T05:45:10.460 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-22T05:45:10.460 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.allow_ingest_behind: 0 2024-03-22T05:45:10.460 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.two_write_queues: 0 2024-03-22T05:45:10.460 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.manual_wal_flush: 0 2024-03-22T05:45:10.461 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.wal_compression: 0 2024-03-22T05:45:10.461 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.atomic_flush: 0 2024-03-22T05:45:10.461 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-22T05:45:10.461 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-22T05:45:10.461 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-22T05:45:10.461 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.log_readahead_size: 0 2024-03-22T05:45:10.461 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-22T05:45:10.461 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.best_efforts_recovery: 0 2024-03-22T05:45:10.461 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-22T05:45:10.462 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-22T05:45:10.462 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.allow_data_in_errors: 0 2024-03-22T05:45:10.462 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.db_host_id: __hostname__ 2024-03-22T05:45:10.462 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-22T05:45:10.462 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.max_background_jobs: 2 2024-03-22T05:45:10.462 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.max_background_compactions: -1 2024-03-22T05:45:10.462 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.max_subcompactions: 1 2024-03-22T05:45:10.462 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-22T05:45:10.463 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-22T05:45:10.463 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-22T05:45:10.463 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.max_total_wal_size: 0 2024-03-22T05:45:10.463 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-22T05:45:10.463 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-22T05:45:10.463 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-22T05:45:10.463 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-22T05:45:10.463 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.max_open_files: -1 2024-03-22T05:45:10.463 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.bytes_per_sync: 0 2024-03-22T05:45:10.463 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-22T05:45:10.464 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-22T05:45:10.464 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.compaction_readahead_size: 0 2024-03-22T05:45:10.464 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.max_background_flushes: -1 2024-03-22T05:45:10.464 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Compression algorithms supported: 2024-03-22T05:45:10.464 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-22T05:45:10.464 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: kZSTD supported: 0 2024-03-22T05:45:10.464 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: kXpressCompression supported: 0 2024-03-22T05:45:10.464 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: kLZ4HCCompression supported: 1 2024-03-22T05:45:10.464 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: kZlibCompression supported: 1 2024-03-22T05:45:10.464 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: kSnappyCompression supported: 1 2024-03-22T05:45:10.464 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: kLZ4Compression supported: 1 2024-03-22T05:45:10.465 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: kBZip2Compression supported: 0 2024-03-22T05:45:10.465 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-22T05:45:10.465 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-22T05:45:10.465 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000009 2024-03-22T05:45:10.465 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-22T05:45:10.465 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-22T05:45:10.465 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.merge_operator: 2024-03-22T05:45:10.465 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.compaction_filter: None 2024-03-22T05:45:10.465 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.compaction_filter_factory: None 2024-03-22T05:45:10.465 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.sst_partitioner_factory: None 2024-03-22T05:45:10.466 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-22T05:45:10.466 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-22T05:45:10.466 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x564509fb9a20) 2024-03-22T05:45:10.466 INFO:journalctl@ceph.mon.c.smithi005.stdout: cache_index_and_filter_blocks: 1 2024-03-22T05:45:10.466 INFO:journalctl@ceph.mon.c.smithi005.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-22T05:45:10.466 INFO:journalctl@ceph.mon.c.smithi005.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-22T05:45:10.466 INFO:journalctl@ceph.mon.c.smithi005.stdout: pin_top_level_index_and_filter: 1 2024-03-22T05:45:10.466 INFO:journalctl@ceph.mon.c.smithi005.stdout: index_type: 0 2024-03-22T05:45:10.466 INFO:journalctl@ceph.mon.c.smithi005.stdout: data_block_index_type: 0 2024-03-22T05:45:10.466 INFO:journalctl@ceph.mon.c.smithi005.stdout: index_shortening: 1 2024-03-22T05:45:10.466 INFO:journalctl@ceph.mon.c.smithi005.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-22T05:45:10.467 INFO:journalctl@ceph.mon.c.smithi005.stdout: checksum: 4 2024-03-22T05:45:10.467 INFO:journalctl@ceph.mon.c.smithi005.stdout: no_block_cache: 0 2024-03-22T05:45:10.467 INFO:journalctl@ceph.mon.c.smithi005.stdout: block_cache: 0x564509fbd090 2024-03-22T05:45:10.467 INFO:journalctl@ceph.mon.c.smithi005.stdout: block_cache_name: BinnedLRUCache 2024-03-22T05:45:10.467 INFO:journalctl@ceph.mon.c.smithi005.stdout: block_cache_options: 2024-03-22T05:45:10.467 INFO:journalctl@ceph.mon.c.smithi005.stdout: capacity : 536870912 2024-03-22T05:45:10.467 INFO:journalctl@ceph.mon.c.smithi005.stdout: num_shard_bits : 4 2024-03-22T05:45:10.467 INFO:journalctl@ceph.mon.c.smithi005.stdout: strict_capacity_limit : 0 2024-03-22T05:45:10.467 INFO:journalctl@ceph.mon.c.smithi005.stdout: high_pri_pool_ratio: 0.000 2024-03-22T05:45:10.467 INFO:journalctl@ceph.mon.c.smithi005.stdout: block_cache_compressed: (nil) 2024-03-22T05:45:10.467 INFO:journalctl@ceph.mon.c.smithi005.stdout: persistent_cache: (nil) 2024-03-22T05:45:10.468 INFO:journalctl@ceph.mon.c.smithi005.stdout: block_size: 4096 2024-03-22T05:45:10.468 INFO:journalctl@ceph.mon.c.smithi005.stdout: block_size_deviation: 10 2024-03-22T05:45:10.468 INFO:journalctl@ceph.mon.c.smithi005.stdout: block_restart_interval: 16 2024-03-22T05:45:10.468 INFO:journalctl@ceph.mon.c.smithi005.stdout: index_block_restart_interval: 1 2024-03-22T05:45:10.468 INFO:journalctl@ceph.mon.c.smithi005.stdout: metadata_block_size: 4096 2024-03-22T05:45:10.468 INFO:journalctl@ceph.mon.c.smithi005.stdout: partition_filters: 0 2024-03-22T05:45:10.468 INFO:journalctl@ceph.mon.c.smithi005.stdout: use_delta_encoding: 1 2024-03-22T05:45:10.468 INFO:journalctl@ceph.mon.c.smithi005.stdout: filter_policy: bloomfilter 2024-03-22T05:45:10.468 INFO:journalctl@ceph.mon.c.smithi005.stdout: whole_key_filtering: 1 2024-03-22T05:45:10.468 INFO:journalctl@ceph.mon.c.smithi005.stdout: verify_compression: 0 2024-03-22T05:45:10.468 INFO:journalctl@ceph.mon.c.smithi005.stdout: read_amp_bytes_per_bit: 0 2024-03-22T05:45:10.468 INFO:journalctl@ceph.mon.c.smithi005.stdout: format_version: 5 2024-03-22T05:45:10.468 INFO:journalctl@ceph.mon.c.smithi005.stdout: enable_index_compression: 1 2024-03-22T05:45:10.468 INFO:journalctl@ceph.mon.c.smithi005.stdout: block_align: 0 2024-03-22T05:45:10.469 INFO:journalctl@ceph.mon.c.smithi005.stdout: max_auto_readahead_size: 262144 2024-03-22T05:45:10.469 INFO:journalctl@ceph.mon.c.smithi005.stdout: prepopulate_block_cache: 0 2024-03-22T05:45:10.469 INFO:journalctl@ceph.mon.c.smithi005.stdout: initial_auto_readahead_size: 8192 2024-03-22T05:45:10.469 INFO:journalctl@ceph.mon.c.smithi005.stdout: num_file_reads_for_auto_readahead: 2 2024-03-22T05:45:10.469 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.write_buffer_size: 33554432 2024-03-22T05:45:10.469 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.max_write_buffer_number: 2 2024-03-22T05:45:10.469 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.compression: NoCompression 2024-03-22T05:45:10.469 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.bottommost_compression: Disabled 2024-03-22T05:45:10.469 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.prefix_extractor: nullptr 2024-03-22T05:45:10.469 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-22T05:45:10.469 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.num_levels: 7 2024-03-22T05:45:10.469 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-22T05:45:10.469 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-22T05:45:10.469 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-22T05:45:10.469 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-22T05:45:10.470 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-22T05:45:10.470 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-22T05:45:10.470 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-22T05:45:10.470 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-22T05:45:10.470 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-22T05:45:10.470 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-22T05:45:10.470 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-22T05:45:10.470 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-22T05:45:10.470 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-22T05:45:10.470 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.compression_opts.level: 32767 2024-03-22T05:45:10.470 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.compression_opts.strategy: 0 2024-03-22T05:45:10.470 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-22T05:45:10.470 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-22T05:45:10.470 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-22T05:45:10.471 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-22T05:45:10.471 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.compression_opts.enabled: false 2024-03-22T05:45:10.471 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-22T05:45:10.471 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-22T05:45:10.471 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-22T05:45:10.471 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-22T05:45:10.471 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.target_file_size_base: 67108864 2024-03-22T05:45:10.471 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-22T05:45:10.471 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-22T05:45:10.471 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-22T05:45:10.471 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-22T05:45:10.471 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-22T05:45:10.471 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-22T05:45:10.471 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-22T05:45:10.471 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-22T05:45:10.471 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-22T05:45:10.472 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-22T05:45:10.472 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-22T05:45:10.472 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-22T05:45:10.472 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-22T05:45:10.472 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-22T05:45:10.472 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.arena_block_size: 1048576 2024-03-22T05:45:10.472 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-22T05:45:10.472 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-22T05:45:10.472 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.disable_auto_compactions: 0 2024-03-22T05:45:10.472 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-22T05:45:10.472 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-22T05:45:10.472 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-22T05:45:10.472 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-22T05:45:10.472 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-22T05:45:10.472 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-22T05:45:10.473 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-22T05:45:10.473 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-22T05:45:10.473 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-22T05:45:10.473 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-22T05:45:10.473 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.table_properties_collectors: 2024-03-22T05:45:10.473 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.inplace_update_support: 0 2024-03-22T05:45:10.473 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-22T05:45:10.473 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-22T05:45:10.473 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-22T05:45:10.473 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-22T05:45:10.473 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.bloom_locality: 0 2024-03-22T05:45:10.473 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.max_successive_merges: 0 2024-03-22T05:45:10.473 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-22T05:45:10.473 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.paranoid_file_checks: 0 2024-03-22T05:45:10.474 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.force_consistency_checks: 1 2024-03-22T05:45:10.474 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.report_bg_io_stats: 0 2024-03-22T05:45:10.474 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.ttl: 2592000 2024-03-22T05:45:10.474 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-22T05:45:10.474 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-22T05:45:10.474 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-22T05:45:10.474 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.enable_blob_files: false 2024-03-22T05:45:10.474 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.min_blob_size: 0 2024-03-22T05:45:10.474 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.blob_file_size: 268435456 2024-03-22T05:45:10.474 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-22T05:45:10.474 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-22T05:45:10.474 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-22T05:45:10.474 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-22T05:45:10.474 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-22T05:45:10.474 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.blob_file_starting_level: 0 2024-03-22T05:45:10.474 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-22T05:45:10.475 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 60.sst 2024-03-22T05:45:10.475 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-22T05:45:10.475 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 62, last_sequence is 32223, log_number is 58,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-22T05:45:10.475 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 58 2024-03-22T05:45:10.475 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 967946cf-0413-42bf-a64d-b7ecf40eb3bb 2024-03-22T05:45:10.475 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711086310215602, "job": 1, "event": "recovery_started", "wal_files": [58]} 2024-03-22T05:45:10.475 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #58 mode 2 2024-03-22T05:45:10.475 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711086310222587, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 63, "file_size": 1772539, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 32228, "largest_seqno": 32704, "table_properties": {"data_size": 1769341, "index_size": 1608, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 5444, "raw_average_key_size": 23, "raw_value_size": 1764212, "raw_average_value_size": 7771, "num_data_blocks": 72, "num_entries": 227, "num_filter_entries": 227, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711086310, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "967946cf-0413-42bf-a64d-b7ecf40eb3bb", "db_session_id": "RU8ASWXNDTG7A8J3O3MJ", "orig_file_number": 63, "seqno_to_time_mapping": "N/A"}} 2024-03-22T05:45:10.475 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711086310222686, "job": 1, "event": "recovery_finished"} 2024-03-22T05:45:10.475 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: [db/version_set.cc:5047] Creating manifest 65 2024-03-22T05:45:10.475 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-22T05:45:10.475 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000058.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-22T05:45:10.475 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x56450a0ac000 2024-03-22T05:45:10.476 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: DB pointer 0x56450a096000 2024-03-22T05:45:10.476 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-22T05:45:10.476 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-22T05:45:10.476 INFO:journalctl@ceph.mon.c.smithi005.stdout: ** DB Stats ** 2024-03-22T05:45:10.476 INFO:journalctl@ceph.mon.c.smithi005.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-22T05:45:10.476 INFO:journalctl@ceph.mon.c.smithi005.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-22T05:45:10.476 INFO:journalctl@ceph.mon.c.smithi005.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-22T05:45:10.476 INFO:journalctl@ceph.mon.c.smithi005.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-22T05:45:10.476 INFO:journalctl@ceph.mon.c.smithi005.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-22T05:45:10.476 INFO:journalctl@ceph.mon.c.smithi005.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-22T05:45:10.476 INFO:journalctl@ceph.mon.c.smithi005.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-22T05:45:10.476 INFO:journalctl@ceph.mon.c.smithi005.stdout: 2024-03-22T05:45:10.476 INFO:journalctl@ceph.mon.c.smithi005.stdout: ** Compaction Stats [default] ** 2024-03-22T05:45:10.476 INFO:journalctl@ceph.mon.c.smithi005.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-22T05:45:10.476 INFO:journalctl@ceph.mon.c.smithi005.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-22T05:45:10.477 INFO:journalctl@ceph.mon.c.smithi005.stdout: L0 1/0 1.69 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 513.5 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-03-22T05:45:10.477 INFO:journalctl@ceph.mon.c.smithi005.stdout: L6 1/0 10.14 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-03-22T05:45:10.477 INFO:journalctl@ceph.mon.c.smithi005.stdout: Sum 2/0 11.83 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 513.5 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-03-22T05:45:10.477 INFO:journalctl@ceph.mon.c.smithi005.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 513.5 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-03-22T05:45:10.477 INFO:journalctl@ceph.mon.c.smithi005.stdout: 2024-03-22T05:45:10.477 INFO:journalctl@ceph.mon.c.smithi005.stdout: ** Compaction Stats [default] ** 2024-03-22T05:45:10.477 INFO:journalctl@ceph.mon.c.smithi005.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-22T05:45:10.477 INFO:journalctl@ceph.mon.c.smithi005.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-22T05:45:10.477 INFO:journalctl@ceph.mon.c.smithi005.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 513.5 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-03-22T05:45:10.477 INFO:journalctl@ceph.mon.c.smithi005.stdout: 2024-03-22T05:45:10.477 INFO:journalctl@ceph.mon.c.smithi005.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-22T05:45:10.477 INFO:journalctl@ceph.mon.c.smithi005.stdout: 2024-03-22T05:45:10.477 INFO:journalctl@ceph.mon.c.smithi005.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-22T05:45:10.477 INFO:journalctl@ceph.mon.c.smithi005.stdout: Flush(GB): cumulative 0.002, interval 0.002 2024-03-22T05:45:10.477 INFO:journalctl@ceph.mon.c.smithi005.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-22T05:45:10.477 INFO:journalctl@ceph.mon.c.smithi005.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-22T05:45:10.478 INFO:journalctl@ceph.mon.c.smithi005.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-22T05:45:10.478 INFO:journalctl@ceph.mon.c.smithi005.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-22T05:45:10.478 INFO:journalctl@ceph.mon.c.smithi005.stdout: Cumulative compaction: 0.00 GB write, 181.59 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-22T05:45:10.478 INFO:journalctl@ceph.mon.c.smithi005.stdout: Interval compaction: 0.00 GB write, 181.59 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-22T05:45:10.478 INFO:journalctl@ceph.mon.c.smithi005.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-22T05:45:10.478 INFO:journalctl@ceph.mon.c.smithi005.stdout: Block cache BinnedLRUCache@0x564509fbd090#2 capacity: 512.00 MB usage: 2.48 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-22T05:45:10.478 INFO:journalctl@ceph.mon.c.smithi005.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.66 KB,0.00012517%) IndexBlock(1,1.83 KB,0.000348687%) Misc(1,0.00 KB,0%) 2024-03-22T05:45:10.478 INFO:journalctl@ceph.mon.c.smithi005.stdout: 2024-03-22T05:45:10.478 INFO:journalctl@ceph.mon.c.smithi005.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-22T05:45:10.478 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: starting mon.c rank 1 at public addrs [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] at bind addrs [v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid f59f36e2-e80a-11ee-95cd-87774f69a715 2024-03-22T05:45:10.478 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: mon.c@-1(???) e3 preinit fsid f59f36e2-e80a-11ee-95cd-87774f69a715 2024-03-22T05:45:10.478 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: mon.c@-1(???).mds e1 new map 2024-03-22T05:45:10.478 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: mon.c@-1(???).mds e1 print_map 2024-03-22T05:45:10.479 INFO:journalctl@ceph.mon.c.smithi005.stdout: e1 2024-03-22T05:45:10.479 INFO:journalctl@ceph.mon.c.smithi005.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-22T05:45:10.479 INFO:journalctl@ceph.mon.c.smithi005.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-22T05:45:10.479 INFO:journalctl@ceph.mon.c.smithi005.stdout: legacy client fscid: -1 2024-03-22T05:45:10.479 INFO:journalctl@ceph.mon.c.smithi005.stdout: 2024-03-22T05:45:10.479 INFO:journalctl@ceph.mon.c.smithi005.stdout: No filesystems configured 2024-03-22T05:45:10.479 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: mon.c@-1(???).osd e105 crush map has features 3314933000854323200, adjusting msgr requires 2024-03-22T05:45:10.479 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: mon.c@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-03-22T05:45:10.479 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: mon.c@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-03-22T05:45:10.479 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: mon.c@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-03-22T05:45:10.479 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 ceph-mon[89904]: mon.c@-1(???).paxosservice(auth 1..28) refresh upgraded, format 0 -> 3 2024-03-22T05:45:10.479 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 bash[89884]: 708c72ff74f5b841a258f70d64a4f638b6da5c5ea37a4ba779c0151f63a841c5 2024-03-22T05:45:10.479 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:10 smithi005 systemd[1]: Started Ceph mon.c for f59f36e2-e80a-11ee-95cd-87774f69a715. 2024-03-22T05:45:11.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:11 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-22T05:45:11.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:11 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T05:45:11.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:11 smithi005 ceph-mon[84705]: mon.c calling monitor election 2024-03-22T05:45:11.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:11 smithi005 ceph-mon[84705]: mon.a calling monitor election 2024-03-22T05:45:11.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:11 smithi005 ceph-mon[84705]: mon.b calling monitor election 2024-03-22T05:45:11.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:11 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T05:45:11.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:11 smithi005 ceph-mon[84705]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-22T05:45:11.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:11 smithi005 ceph-mon[84705]: monmap e4: 3 mons at {a=[v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0],b=[v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0],c=[v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-22T05:45:11.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:11 smithi005 ceph-mon[84705]: fsmap 2024-03-22T05:45:11.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:11 smithi005 ceph-mon[84705]: osdmap e105: 8 total, 8 up, 8 in 2024-03-22T05:45:11.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:11 smithi005 ceph-mon[84705]: mgrmap e50: y(active, since 18s), standbys: x 2024-03-22T05:45:11.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:11 smithi005 ceph-mon[84705]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-22T05:45:11.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:11 smithi005 ceph-mon[84705]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-22T05:45:11.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:11 smithi005 ceph-mon[84705]: daemon iscsi.foo.smithi005.mcdwbp on smithi005 is in unknown state 2024-03-22T05:45:11.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:11 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='' 2024-03-22T05:45:11.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:11 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:11.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:11 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:45:11.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:11 smithi005 ceph-mon[84705]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 985 B/s rd, 0 op/s 2024-03-22T05:45:11.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:11 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-22T05:45:11.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:11 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T05:45:11.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:11 smithi005 ceph-mon[89904]: mon.c calling monitor election 2024-03-22T05:45:11.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:11 smithi005 ceph-mon[89904]: mon.a calling monitor election 2024-03-22T05:45:11.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:11 smithi005 ceph-mon[89904]: mon.b calling monitor election 2024-03-22T05:45:11.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:11 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T05:45:11.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:11 smithi005 ceph-mon[89904]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-22T05:45:11.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:11 smithi005 ceph-mon[89904]: monmap e4: 3 mons at {a=[v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0],b=[v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0],c=[v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-22T05:45:11.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:11 smithi005 ceph-mon[89904]: fsmap 2024-03-22T05:45:11.705 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:11 smithi005 ceph-mon[89904]: osdmap e105: 8 total, 8 up, 8 in 2024-03-22T05:45:11.705 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:11 smithi005 ceph-mon[89904]: mgrmap e50: y(active, since 18s), standbys: x 2024-03-22T05:45:11.705 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:11 smithi005 ceph-mon[89904]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-22T05:45:11.705 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:11 smithi005 ceph-mon[89904]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-22T05:45:11.705 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:11 smithi005 ceph-mon[89904]: daemon iscsi.foo.smithi005.mcdwbp on smithi005 is in unknown state 2024-03-22T05:45:11.705 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:11 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='' 2024-03-22T05:45:11.705 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:11 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:11.705 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:11 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:45:11.705 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:11 smithi005 ceph-mon[89904]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 985 B/s rd, 0 op/s 2024-03-22T05:45:12.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:11 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-22T05:45:12.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:11 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-22T05:45:12.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:11 smithi186 ceph-mon[67903]: mon.c calling monitor election 2024-03-22T05:45:12.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:11 smithi186 ceph-mon[67903]: mon.a calling monitor election 2024-03-22T05:45:12.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:11 smithi186 ceph-mon[67903]: mon.b calling monitor election 2024-03-22T05:45:12.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:11 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-22T05:45:12.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:11 smithi186 ceph-mon[67903]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-22T05:45:12.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:11 smithi186 ceph-mon[67903]: monmap e4: 3 mons at {a=[v2:172.21.15.5:3300/0,v1:172.21.15.5:6789/0],b=[v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0],c=[v2:172.21.15.5:3301/0,v1:172.21.15.5:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-22T05:45:12.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:11 smithi186 ceph-mon[67903]: fsmap 2024-03-22T05:45:12.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:11 smithi186 ceph-mon[67903]: osdmap e105: 8 total, 8 up, 8 in 2024-03-22T05:45:12.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:11 smithi186 ceph-mon[67903]: mgrmap e50: y(active, since 18s), standbys: x 2024-03-22T05:45:12.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:11 smithi186 ceph-mon[67903]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-22T05:45:12.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:11 smithi186 ceph-mon[67903]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-22T05:45:12.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:11 smithi186 ceph-mon[67903]: daemon iscsi.foo.smithi005.mcdwbp on smithi005 is in unknown state 2024-03-22T05:45:12.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:11 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='' 2024-03-22T05:45:12.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:11 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:12.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:11 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:45:12.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:11 smithi186 ceph-mon[67903]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 985 B/s rd, 0 op/s 2024-03-22T05:45:14.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:13 smithi005 ceph-mon[89904]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 985 B/s rd, 0 op/s 2024-03-22T05:45:14.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:13 smithi005 ceph-mon[84705]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 985 B/s rd, 0 op/s 2024-03-22T05:45:14.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:13 smithi186 ceph-mon[67903]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 985 B/s rd, 0 op/s 2024-03-22T05:45:15.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:14 smithi005 ceph-mon[89904]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:45:15.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:14 smithi005 ceph-mon[84705]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:45:15.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:14 smithi186 ceph-mon[67903]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:45:15.702 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:15 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:45:15.230+0000 7f7f3e571700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-22T05:45:16.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:16 smithi005 ceph-mon[89904]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:45:16.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:16 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:16.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:16 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:16.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:16 smithi005 ceph-mon[84705]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:45:16.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:16 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:16.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:16 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:16.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:16 smithi186 ceph-mon[67903]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:45:16.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:16 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:16.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:16 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:17.190 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:16 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:45:16] "GET /metrics HTTP/1.1" 200 37998 "" "Prometheus/2.43.0" 2024-03-22T05:45:17.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:17 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:17.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:17 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:17.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:17 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:17.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:17 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:17.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:17 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:17.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:17 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:18.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:18 smithi005 ceph-mon[84705]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:45:18.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:18 smithi005 ceph-mon[89904]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:45:18.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:18 smithi186 ceph-mon[67903]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:45:20.512 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:20 smithi005 ceph-mon[84705]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:45:20.512 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:20 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:20.512 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:20 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:20.512 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:20 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:45:20.512 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:20 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:45:20.512 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:20 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:20.512 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:20 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-22T05:45:20.513 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:20 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-22T05:45:20.513 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:20 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:45:20.513 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:20 smithi005 ceph-mon[89904]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:45:20.513 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:20 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:20.513 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:20 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:20.513 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:20 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:45:20.513 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:20 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:45:20.513 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:20 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:20.513 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:20 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-22T05:45:20.513 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:20 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-22T05:45:20.513 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:20 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:45:20.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:20 smithi186 ceph-mon[67903]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:45:20.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:20 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:20.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:20 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:20.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:20 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:45:20.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:20 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:45:20.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:20 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:20.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:20 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-22T05:45:20.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:20 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-22T05:45:20.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:20 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:45:21.517 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:21 smithi005 ceph-mon[89904]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 932 B/s rd, 0 op/s 2024-03-22T05:45:21.517 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:21 smithi005 ceph-mon[89904]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:45:21.517 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:21 smithi005 ceph-mon[89904]: Reconfiguring mon.a (monmap changed)... 2024-03-22T05:45:21.518 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:21 smithi005 ceph-mon[89904]: Reconfiguring daemon mon.a on smithi005 2024-03-22T05:45:21.518 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:21 smithi005 ceph-mon[84705]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 932 B/s rd, 0 op/s 2024-03-22T05:45:21.518 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:21 smithi005 ceph-mon[84705]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:45:21.518 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:21 smithi005 ceph-mon[84705]: Reconfiguring mon.a (monmap changed)... 2024-03-22T05:45:21.518 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:21 smithi005 ceph-mon[84705]: Reconfiguring daemon mon.a on smithi005 2024-03-22T05:45:21.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:21 smithi186 ceph-mon[67903]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 932 B/s rd, 0 op/s 2024-03-22T05:45:21.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:21 smithi186 ceph-mon[67903]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:45:21.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:21 smithi186 ceph-mon[67903]: Reconfiguring mon.a (monmap changed)... 2024-03-22T05:45:21.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:21 smithi186 ceph-mon[67903]: Reconfiguring daemon mon.a on smithi005 2024-03-22T05:45:22.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:22 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:22.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:22 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:22.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:22 smithi005 ceph-mon[84705]: Reconfiguring mgr.y (monmap changed)... 2024-03-22T05:45:22.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:22 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-22T05:45:22.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:22 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-22T05:45:22.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:22 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-22T05:45:22.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:22 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:45:22.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:22 smithi005 ceph-mon[84705]: Reconfiguring daemon mgr.y on smithi005 2024-03-22T05:45:22.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:22 smithi005 ceph-mon[84705]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:45:22.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:22 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:22.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:22 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:22.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:22 smithi005 ceph-mon[89904]: Reconfiguring mgr.y (monmap changed)... 2024-03-22T05:45:22.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:22 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-22T05:45:22.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:22 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-22T05:45:22.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:22 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-22T05:45:22.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:22 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:45:22.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:22 smithi005 ceph-mon[89904]: Reconfiguring daemon mgr.y on smithi005 2024-03-22T05:45:22.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:22 smithi005 ceph-mon[89904]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:45:22.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:22 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:22.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:22 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:22.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:22 smithi186 ceph-mon[67903]: Reconfiguring mgr.y (monmap changed)... 2024-03-22T05:45:22.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:22 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-22T05:45:22.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:22 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-22T05:45:22.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:22 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-22T05:45:22.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:22 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:45:22.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:22 smithi186 ceph-mon[67903]: Reconfiguring daemon mgr.y on smithi005 2024-03-22T05:45:22.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:22 smithi186 ceph-mon[67903]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:45:23.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:23 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:45:23.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:23 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:23.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:23 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:23.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:23 smithi005 ceph-mon[89904]: Reconfiguring mon.c (monmap changed)... 2024-03-22T05:45:23.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:23 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-22T05:45:23.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:23 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-22T05:45:23.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:23 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:45:23.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:23 smithi005 ceph-mon[89904]: Reconfiguring daemon mon.c on smithi005 2024-03-22T05:45:23.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:23 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:45:23.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:23 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:23.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:23 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:23.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:23 smithi005 ceph-mon[84705]: Reconfiguring mon.c (monmap changed)... 2024-03-22T05:45:23.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:23 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-22T05:45:23.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:23 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-22T05:45:23.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:23 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:45:23.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:23 smithi005 ceph-mon[84705]: Reconfiguring daemon mon.c on smithi005 2024-03-22T05:45:23.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:23 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:45:23.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:23 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:23.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:23 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:23.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:23 smithi186 ceph-mon[67903]: Reconfiguring mon.c (monmap changed)... 2024-03-22T05:45:23.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:23 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-22T05:45:23.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:23 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-22T05:45:23.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:23 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:45:23.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:23 smithi186 ceph-mon[67903]: Reconfiguring daemon mon.c on smithi005 2024-03-22T05:45:24.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:24 smithi005 ceph-mon[84705]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:45:24.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:24 smithi005 ceph-mon[84705]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:45:24.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:24 smithi005 ceph-mon[89904]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:45:24.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:24 smithi005 ceph-mon[89904]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:45:24.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:24 smithi186 ceph-mon[67903]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:45:24.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:24 smithi186 ceph-mon[67903]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:45:26.181 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:25 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:26.181 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:25 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:26.181 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:25 smithi005 ceph-mon[89904]: Reconfiguring osd.0 (monmap changed)... 2024-03-22T05:45:26.181 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:25 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-22T05:45:26.181 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:25 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:45:26.181 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:25 smithi005 ceph-mon[89904]: Reconfiguring daemon osd.0 on smithi005 2024-03-22T05:45:26.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:25 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:26.181 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:25 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:26.182 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:25 smithi005 ceph-mon[84705]: Reconfiguring osd.0 (monmap changed)... 2024-03-22T05:45:26.182 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:25 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-22T05:45:26.182 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:25 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:45:26.182 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:25 smithi005 ceph-mon[84705]: Reconfiguring daemon osd.0 on smithi005 2024-03-22T05:45:26.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:25 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:26.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:25 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:26.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:25 smithi186 ceph-mon[67903]: Reconfiguring osd.0 (monmap changed)... 2024-03-22T05:45:26.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:25 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-22T05:45:26.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:25 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:45:26.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:25 smithi186 ceph-mon[67903]: Reconfiguring daemon osd.0 on smithi005 2024-03-22T05:45:27.190 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:26 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:45:26] "GET /metrics HTTP/1.1" 200 38004 "" "Prometheus/2.43.0" 2024-03-22T05:45:27.190 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:26 smithi005 ceph-mon[84705]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:45:27.190 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:26 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:27.190 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:26 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:27.191 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:26 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-22T05:45:27.191 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:26 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:45:27.191 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:26 smithi005 ceph-mon[89904]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:45:27.191 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:26 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:27.191 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:26 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:27.191 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:26 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-22T05:45:27.191 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:26 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:45:27.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:26 smithi186 ceph-mon[67903]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:45:27.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:26 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:27.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:26 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:27.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:26 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-22T05:45:27.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:26 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:45:28.146 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:27 smithi005 ceph-mon[84705]: Reconfiguring osd.1 (monmap changed)... 2024-03-22T05:45:28.146 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:27 smithi005 ceph-mon[84705]: Reconfiguring daemon osd.1 on smithi005 2024-03-22T05:45:28.146 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:27 smithi005 ceph-mon[89904]: Reconfiguring osd.1 (monmap changed)... 2024-03-22T05:45:28.146 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:27 smithi005 ceph-mon[89904]: Reconfiguring daemon osd.1 on smithi005 2024-03-22T05:45:28.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:27 smithi186 ceph-mon[67903]: Reconfiguring osd.1 (monmap changed)... 2024-03-22T05:45:28.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:27 smithi186 ceph-mon[67903]: Reconfiguring daemon osd.1 on smithi005 2024-03-22T05:45:29.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:29 smithi005 ceph-mon[89904]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 570 B/s rd, 0 op/s 2024-03-22T05:45:29.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:29 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:29.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:29 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:29.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:29 smithi005 ceph-mon[89904]: Reconfiguring osd.2 (monmap changed)... 2024-03-22T05:45:29.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:29 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-22T05:45:29.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:29 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:45:29.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:29 smithi005 ceph-mon[89904]: Reconfiguring daemon osd.2 on smithi005 2024-03-22T05:45:29.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:29 smithi005 ceph-mon[84705]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 570 B/s rd, 0 op/s 2024-03-22T05:45:29.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:29 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:29.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:29 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:29.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:29 smithi005 ceph-mon[84705]: Reconfiguring osd.2 (monmap changed)... 2024-03-22T05:45:29.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:29 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-22T05:45:29.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:29 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:45:29.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:29 smithi005 ceph-mon[84705]: Reconfiguring daemon osd.2 on smithi005 2024-03-22T05:45:29.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:29 smithi186 ceph-mon[67903]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 570 B/s rd, 0 op/s 2024-03-22T05:45:29.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:29 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:29.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:29 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:29.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:29 smithi186 ceph-mon[67903]: Reconfiguring osd.2 (monmap changed)... 2024-03-22T05:45:29.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:29 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-22T05:45:29.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:29 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:45:29.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:29 smithi186 ceph-mon[67903]: Reconfiguring daemon osd.2 on smithi005 2024-03-22T05:45:31.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:30 smithi005 ceph-mon[84705]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:45:31.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:30 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:31.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:30 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:31.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:30 smithi005 ceph-mon[84705]: Reconfiguring osd.3 (monmap changed)... 2024-03-22T05:45:31.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:30 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-22T05:45:31.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:30 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:45:31.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:30 smithi005 ceph-mon[84705]: Reconfiguring daemon osd.3 on smithi005 2024-03-22T05:45:31.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:30 smithi005 ceph-mon[89904]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:45:31.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:30 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:31.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:30 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:31.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:30 smithi005 ceph-mon[89904]: Reconfiguring osd.3 (monmap changed)... 2024-03-22T05:45:31.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:30 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-22T05:45:31.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:30 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:45:31.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:30 smithi005 ceph-mon[89904]: Reconfiguring daemon osd.3 on smithi005 2024-03-22T05:45:31.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:30 smithi186 ceph-mon[67903]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:45:31.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:30 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:31.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:30 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:31.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:30 smithi186 ceph-mon[67903]: Reconfiguring osd.3 (monmap changed)... 2024-03-22T05:45:31.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:30 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-22T05:45:31.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:30 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:45:31.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:30 smithi186 ceph-mon[67903]: Reconfiguring daemon osd.3 on smithi005 2024-03-22T05:45:32.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:32 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:32.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:32 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:32.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:32 smithi005 ceph-mon[89904]: Reconfiguring rgw.foo.smithi005.exdhqu (monmap changed)... 2024-03-22T05:45:32.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:32 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi005.exdhqu", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-22T05:45:32.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:32 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi005.exdhqu", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-22T05:45:32.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:32 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:45:32.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:32 smithi005 ceph-mon[89904]: Reconfiguring daemon rgw.foo.smithi005.exdhqu on smithi005 2024-03-22T05:45:32.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:32 smithi005 ceph-mon[89904]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:45:32.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:32 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:32.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:32 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:32.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:32 smithi005 ceph-mon[84705]: Reconfiguring rgw.foo.smithi005.exdhqu (monmap changed)... 2024-03-22T05:45:32.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:32 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi005.exdhqu", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-22T05:45:32.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:32 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi005.exdhqu", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-22T05:45:32.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:32 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:45:32.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:32 smithi005 ceph-mon[84705]: Reconfiguring daemon rgw.foo.smithi005.exdhqu on smithi005 2024-03-22T05:45:32.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:32 smithi005 ceph-mon[84705]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:45:33.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:32 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:33.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:32 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:33.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:32 smithi186 ceph-mon[67903]: Reconfiguring rgw.foo.smithi005.exdhqu (monmap changed)... 2024-03-22T05:45:33.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:32 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi005.exdhqu", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-22T05:45:33.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:32 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi005.exdhqu", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-22T05:45:33.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:32 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:45:33.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:32 smithi186 ceph-mon[67903]: Reconfiguring daemon rgw.foo.smithi005.exdhqu on smithi005 2024-03-22T05:45:33.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:32 smithi186 ceph-mon[67903]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:45:34.649 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:34 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:34.649 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:34 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:34.649 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:34 smithi005 ceph-mon[84705]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:45:34.649 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:34 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:45:34.649 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:34 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:45:34.649 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:34 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:45:34.649 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:34 smithi005 ceph-mon[84705]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:45:34.649 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:34 smithi005 ceph-mon[84705]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:45:34.650 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:34 smithi005 ceph-mon[84705]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:45:34.650 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:34 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:34.650 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:34 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:34.650 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:34 smithi005 ceph-mon[89904]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:45:34.650 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:34 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:45:34.650 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:34 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:45:34.650 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:34 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:45:34.650 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:34 smithi005 ceph-mon[89904]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:45:34.650 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:34 smithi005 ceph-mon[89904]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:45:34.650 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:34 smithi005 ceph-mon[89904]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:45:34.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:34 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:34.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:34 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:34.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:34 smithi186 ceph-mon[67903]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:45:34.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:34 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:45:34.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:34 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:45:34.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:34 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:45:34.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:34 smithi186 ceph-mon[67903]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:45:34.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:34 smithi186 ceph-mon[67903]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:45:34.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:34 smithi186 ceph-mon[67903]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:45:35.202 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:45:35.108+0000 7f7f0ecc9700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:45:35.202 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:45:35.202 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:45:35.203 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:45:35.203 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:45:35.203 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:45:35.203 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:45:35.203 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:45:35.203 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:45:35.203 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:45:35.203 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:45:35.203 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:45:35.203 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:45:35.203 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:45:35.203 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:45:35.204 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:45:35.204 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:45:35.204 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:45:35.204 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:45:35.205 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:45:35.205 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:45:35.205 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:45:35.205 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:45:35.205 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:45:35.205 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:45:35.205 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:45:35.205 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:45:35.205 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:45:35.205 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:45:35.205 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return future.result(timeout) 2024-03-22T05:45:35.206 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:45:35.206 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.__get_result() 2024-03-22T05:45:35.206 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:45:35.206 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: raise self._exception 2024-03-22T05:45:35.206 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:45:35.206 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: config_blobs=daemon_spec.final_config, 2024-03-22T05:45:35.206 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:45:35.206 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:45:35.207 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:45:35.207 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:45:35.207 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:45:35.207 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:45:35.207 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:45:35.207 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:45:35.207 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:45:35.207 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:45:35.207 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:45:35.207 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:45:35.207 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:45:35.207 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:45:35.208 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:45:35.208 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:45:35.208 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:45:35.208 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:45:35.208 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:45:35.208 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:45:35.208 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:45:35.208 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:45:35.208 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:45:35.692 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:35 smithi186 ceph-mon[67903]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:45:35.692 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:45:35.692 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:45:35.692 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:45:35.692 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:45:35.692 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:45:35.693 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:45:35.693 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:45:35.693 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:45:35.693 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:45:35.693 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:45:35.693 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:45:35.693 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:45:35.693 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:45:35.693 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:45:35.693 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:45:35.693 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:45:35.693 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:45:35.693 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:45:35.693 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:45:35.693 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:45:35.694 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:45:35.694 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:45:35.694 INFO:journalctl@ceph.mon.b.smithi186.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:45:35.694 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:45:35.694 INFO:journalctl@ceph.mon.b.smithi186.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:45:35.694 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:45:35.694 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:45:35.694 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:45:35.694 INFO:journalctl@ceph.mon.b.smithi186.stdout: return future.result(timeout) 2024-03-22T05:45:35.694 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:45:35.694 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.__get_result() 2024-03-22T05:45:35.694 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:45:35.694 INFO:journalctl@ceph.mon.b.smithi186.stdout: raise self._exception 2024-03-22T05:45:35.694 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:45:35.695 INFO:journalctl@ceph.mon.b.smithi186.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:45:35.695 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:45:35.695 INFO:journalctl@ceph.mon.b.smithi186.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:45:35.695 INFO:journalctl@ceph.mon.b.smithi186.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:45:35.695 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:45:35.695 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:45:35.695 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:45:35.695 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:45:35.695 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:45:35.695 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:45:35.695 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:45:35.695 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:45:35.695 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:45:35.695 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:45:35.695 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:45:35.696 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:45:35.696 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:45:35.696 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:45:35.696 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:45:35.696 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:45:35.696 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:45:35.696 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:45:35.696 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:45:35.696 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:45:35.696 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:35 smithi186 ceph-mon[67903]: Reconfiguring mon.b (monmap changed)... 2024-03-22T05:45:35.696 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:35 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-22T05:45:35.696 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:35 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-22T05:45:35.696 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:35 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:45:35.696 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:35 smithi186 ceph-mon[67903]: Reconfiguring daemon mon.b on smithi186 2024-03-22T05:45:35.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:35 smithi005 ceph-mon[84705]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:45:35.702 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:45:35.702 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:45:35.702 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:45:35.702 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:45:35.702 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:45:35.702 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:45:35.702 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:45:35.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:45:35.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:45:35.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:45:35.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:45:35.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:45:35.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:45:35.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:45:35.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:45:35.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:45:35.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:45:35.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:45:35.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:45:35.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:45:35.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:45:35.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:45:35.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:45:35.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:45:35.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:45:35.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:45:35.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:45:35.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:45:35.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: return future.result(timeout) 2024-03-22T05:45:35.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:45:35.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.__get_result() 2024-03-22T05:45:35.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:45:35.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: raise self._exception 2024-03-22T05:45:35.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:45:35.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:45:35.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:45:35.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:45:35.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:45:35.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:45:35.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:45:35.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:45:35.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:45:35.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:45:35.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:45:35.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:45:35.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:45:35.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:45:35.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:45:35.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:45:35.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:45:35.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:45:35.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:45:35.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:45:35.706 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:35 smithi005 ceph-mon[89904]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:45:35.706 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:45:35.706 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:45:35.706 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:45:35.706 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:45:35.706 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:45:35.706 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:45:35.706 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:45:35.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:45:35.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:45:35.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:45:35.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:45:35.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:45:35.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:45:35.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:45:35.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:45:35.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:45:35.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:45:35.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:45:35.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:45:35.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:45:35.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:45:35.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:45:35.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:45:35.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:45:35.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:45:35.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:45:35.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:45:35.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:45:35.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: return future.result(timeout) 2024-03-22T05:45:35.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:45:35.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.__get_result() 2024-03-22T05:45:35.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:45:35.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: raise self._exception 2024-03-22T05:45:35.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:45:35.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:45:35.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:45:35.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:45:35.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:45:35.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:45:35.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:45:35.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:45:35.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:45:35.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:45:35.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:45:35.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:45:35.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:45:35.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:45:35.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:45:35.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:45:35.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:45:35.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:45:35.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:45:35.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:45:35.710 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:45:35.710 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:45:35.710 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:45:35.710 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:45:35.710 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:45:35.710 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:35 smithi005 ceph-mon[84705]: Reconfiguring mon.b (monmap changed)... 2024-03-22T05:45:35.710 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:35 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-22T05:45:35.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:35 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-22T05:45:35.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:35 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:45:35.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:35 smithi005 ceph-mon[84705]: Reconfiguring daemon mon.b on smithi186 2024-03-22T05:45:35.711 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:45:35.711 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:45:35.711 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:45:35.711 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:45:35.711 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:45:35.711 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:35 smithi005 ceph-mon[89904]: Reconfiguring mon.b (monmap changed)... 2024-03-22T05:45:35.711 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:35 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-22T05:45:35.711 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:35 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-22T05:45:35.712 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:35 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:45:35.712 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:35 smithi005 ceph-mon[89904]: Reconfiguring daemon mon.b on smithi186 2024-03-22T05:45:36.618 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:36 smithi186 ceph-mon[67903]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:45:36.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:36 smithi005 ceph-mon[84705]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:45:36.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:36 smithi005 ceph-mon[89904]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:45:37.452 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:36 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:45:36] "GET /metrics HTTP/1.1" 200 38004 "" "Prometheus/2.43.0" 2024-03-22T05:45:37.933 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:37 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:37.933 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:37 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:37.933 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:37 smithi186 ceph-mon[67903]: Reconfiguring mgr.x (monmap changed)... 2024-03-22T05:45:37.933 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:37 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-22T05:45:37.933 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:37 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-22T05:45:37.933 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:37 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-22T05:45:37.933 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:37 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:45:37.933 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:37 smithi186 ceph-mon[67903]: Reconfiguring daemon mgr.x on smithi186 2024-03-22T05:45:37.933 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:37 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:45:37.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:37 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:37.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:37 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:37.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:37 smithi005 ceph-mon[84705]: Reconfiguring mgr.x (monmap changed)... 2024-03-22T05:45:37.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:37 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-22T05:45:37.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:37 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-22T05:45:37.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:37 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-22T05:45:37.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:37 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:45:37.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:37 smithi005 ceph-mon[84705]: Reconfiguring daemon mgr.x on smithi186 2024-03-22T05:45:37.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:37 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:45:37.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:37 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:37.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:37 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:37.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:37 smithi005 ceph-mon[89904]: Reconfiguring mgr.x (monmap changed)... 2024-03-22T05:45:37.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:37 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-22T05:45:37.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:37 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-22T05:45:37.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:37 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-22T05:45:37.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:37 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:45:37.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:37 smithi005 ceph-mon[89904]: Reconfiguring daemon mgr.x on smithi186 2024-03-22T05:45:37.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:37 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:45:38.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:38 smithi005 ceph-mon[89904]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:45:38.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:38 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:38.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:38 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:38.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:38 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-22T05:45:38.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:38 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:45:38.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:38 smithi005 ceph-mon[84705]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:45:38.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:38 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:38.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:38 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:38.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:38 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-22T05:45:38.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:38 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:45:39.020 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:38 smithi186 ceph-mon[67903]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:45:39.020 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:38 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:39.020 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:38 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:39.021 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:38 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-22T05:45:39.021 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:38 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:45:39.281 INFO:teuthology.orchestra.run.smithi005.stdout:true 2024-03-22T05:45:39.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:39 smithi005 ceph-mon[89904]: Reconfiguring osd.4 (monmap changed)... 2024-03-22T05:45:39.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:39 smithi005 ceph-mon[89904]: Reconfiguring daemon osd.4 on smithi186 2024-03-22T05:45:39.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:39 smithi005 ceph-mon[89904]: from='client.54168 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:45:39.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:39 smithi005 ceph-mon[84705]: Reconfiguring osd.4 (monmap changed)... 2024-03-22T05:45:39.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:39 smithi005 ceph-mon[84705]: Reconfiguring daemon osd.4 on smithi186 2024-03-22T05:45:39.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:39 smithi005 ceph-mon[84705]: from='client.54168 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:45:39.953 INFO:teuthology.orchestra.run.smithi005.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T05:45:39.953 INFO:teuthology.orchestra.run.smithi005.stdout:alertmanager.a smithi005 *:9093,9094 running (3m) 24s ago 25m 19.7M - 0.25.0 c8568f914cd2 ee99dfc435e8 2024-03-22T05:45:39.953 INFO:teuthology.orchestra.run.smithi005.stdout:grafana.a smithi186 *:3000 running (2m) 42s ago 24m 81.5M - 9.4.12 2bacad6d85d8 16fd57dbbd71 2024-03-22T05:45:39.953 INFO:teuthology.orchestra.run.smithi005.stdout:iscsi.foo.smithi005.mcdwbp smithi005 unknown 24s ago 24m 62.4M - 3.5 e1d6a67b021e 5215e03bf81b 2024-03-22T05:45:39.953 INFO:teuthology.orchestra.run.smithi005.stdout:mgr.x smithi186 *:8443,9283,8765 running (4m) 42s ago 29m 448M - 19.0.0-2299-gea01217f aa461e7c5e7f c556421a73f3 2024-03-22T05:45:39.953 INFO:teuthology.orchestra.run.smithi005.stdout:mgr.y smithi005 *:8443,9283,8765 running (14m) 24s ago 31m 496M - 19.0.0-2299-gea01217f aa461e7c5e7f 62a813eebc0d 2024-03-22T05:45:39.953 INFO:teuthology.orchestra.run.smithi005.stdout:mon.a smithi005 running (58s) 24s ago 31m 42.7M 2048M 19.0.0-2299-gea01217f aa461e7c5e7f 111a2aa24dd9 2024-03-22T05:45:39.953 INFO:teuthology.orchestra.run.smithi005.stdout:mon.b smithi186 running (108s) 42s ago 29m 39.1M 2048M 19.0.0-2299-gea01217f aa461e7c5e7f c1798feac15b 2024-03-22T05:45:39.953 INFO:teuthology.orchestra.run.smithi005.stdout:mon.c smithi005 running (30s) 24s ago 29m 20.1M 2048M 19.0.0-2299-gea01217f aa461e7c5e7f 708c72ff74f5 2024-03-22T05:45:39.953 INFO:teuthology.orchestra.run.smithi005.stdout:node-exporter.a smithi005 *:9100 running (3m) 24s ago 25m 8627k - 1.5.0 0da6a335fe13 f635cd72b9d3 2024-03-22T05:45:39.953 INFO:teuthology.orchestra.run.smithi005.stdout:node-exporter.b smithi186 *:9100 running (3m) 42s ago 25m 15.1M - 1.5.0 0da6a335fe13 d7e0b3d20191 2024-03-22T05:45:39.953 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0 smithi005 running (29m) 24s ago 29m 53.9M 2993M 17.2.0 e1d6a67b021e ef5c7053dd37 2024-03-22T05:45:39.954 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1 smithi005 running (28m) 24s ago 28m 56.7M 2993M 17.2.0 e1d6a67b021e a1e7ee1017b6 2024-03-22T05:45:39.954 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2 smithi005 running (28m) 24s ago 28m 49.0M 2993M 17.2.0 e1d6a67b021e 462ea63ad0e5 2024-03-22T05:45:39.954 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3 smithi005 running (28m) 24s ago 28m 57.0M 2993M 17.2.0 e1d6a67b021e b25432a7239b 2024-03-22T05:45:39.954 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4 smithi186 running (27m) 42s ago 27m 56.4M 3248M 17.2.0 e1d6a67b021e cd963671f340 2024-03-22T05:45:39.954 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5 smithi186 running (27m) 42s ago 27m 51.7M 3248M 17.2.0 e1d6a67b021e f98c87f1a2cd 2024-03-22T05:45:39.954 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6 smithi186 running (26m) 42s ago 26m 52.0M 3248M 17.2.0 e1d6a67b021e 704ac49c68c4 2024-03-22T05:45:39.954 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7 smithi186 running (26m) 42s ago 26m 54.3M 3248M 17.2.0 e1d6a67b021e f415b90edf63 2024-03-22T05:45:39.954 INFO:teuthology.orchestra.run.smithi005.stdout:prometheus.a smithi186 *:9095 running (3m) 42s ago 25m 67.2M - 2.43.0 a07b618ecd1d a5a043588d6d 2024-03-22T05:45:39.954 INFO:teuthology.orchestra.run.smithi005.stdout:rgw.foo.smithi005.exdhqu smithi005 *:8000 running (24m) 24s ago 24m 80.5M - 17.2.0 e1d6a67b021e 6bb27fe11553 2024-03-22T05:45:39.954 INFO:teuthology.orchestra.run.smithi005.stdout:rgw.foo.smithi186.mcgodk smithi186 *:8000 running (24m) 42s ago 24m 78.2M - 17.2.0 e1d6a67b021e 19b5c20a34f7 2024-03-22T05:45:39.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:39 smithi186 ceph-mon[67903]: Reconfiguring osd.4 (monmap changed)... 2024-03-22T05:45:39.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:39 smithi186 ceph-mon[67903]: Reconfiguring daemon osd.4 on smithi186 2024-03-22T05:45:39.964 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:39 smithi186 ceph-mon[67903]: from='client.54168 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:45:40.365 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-03-22T05:45:40.365 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": { 2024-03-22T05:45:40.365 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev)": 3 2024-03-22T05:45:40.365 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:45:40.365 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr": { 2024-03-22T05:45:40.365 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev)": 2 2024-03-22T05:45:40.365 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:45:40.365 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": { 2024-03-22T05:45:40.365 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-22T05:45:40.365 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:45:40.366 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw": { 2024-03-22T05:45:40.366 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-22T05:45:40.366 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:45:40.366 INFO:teuthology.orchestra.run.smithi005.stdout: "overall": { 2024-03-22T05:45:40.366 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 10, 2024-03-22T05:45:40.366 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev)": 5 2024-03-22T05:45:40.366 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-03-22T05:45:40.366 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-03-22T05:45:40.708 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-03-22T05:45:40.709 INFO:teuthology.orchestra.run.smithi005.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9", 2024-03-22T05:45:40.709 INFO:teuthology.orchestra.run.smithi005.stdout: "in_progress": true, 2024-03-22T05:45:40.709 INFO:teuthology.orchestra.run.smithi005.stdout: "which": "Upgrading daemons of type(s) mon on host(s) smithi005", 2024-03-22T05:45:40.709 INFO:teuthology.orchestra.run.smithi005.stdout: "services_complete": [ 2024-03-22T05:45:40.709 INFO:teuthology.orchestra.run.smithi005.stdout: "mon" 2024-03-22T05:45:40.709 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-03-22T05:45:40.709 INFO:teuthology.orchestra.run.smithi005.stdout: "progress": "2/2 daemons upgraded", 2024-03-22T05:45:40.709 INFO:teuthology.orchestra.run.smithi005.stdout: "message": "Currently upgrading mon daemons", 2024-03-22T05:45:40.709 INFO:teuthology.orchestra.run.smithi005.stdout: "is_paused": false 2024-03-22T05:45:40.709 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-03-22T05:45:41.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:41 smithi186 ceph-mon[67903]: from='client.64108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:45:41.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:41 smithi186 ceph-mon[67903]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:45:41.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:41 smithi186 ceph-mon[67903]: from='client.64114 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:45:41.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:41 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:41.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:41 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:41.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:41 smithi186 ceph-mon[67903]: Reconfiguring osd.5 (monmap changed)... 2024-03-22T05:45:41.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:41 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-22T05:45:41.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:41 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:45:41.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:41 smithi186 ceph-mon[67903]: Reconfiguring daemon osd.5 on smithi186 2024-03-22T05:45:41.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:41 smithi186 ceph-mon[67903]: from='client.? 172.21.15.5:0/3136461754' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:45:41.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:41 smithi005 ceph-mon[89904]: from='client.64108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:45:41.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:41 smithi005 ceph-mon[89904]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:45:41.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:41 smithi005 ceph-mon[89904]: from='client.64114 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:45:41.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:41 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:41.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:41 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:41.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:41 smithi005 ceph-mon[89904]: Reconfiguring osd.5 (monmap changed)... 2024-03-22T05:45:41.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:41 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-22T05:45:41.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:41 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:45:41.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:41 smithi005 ceph-mon[89904]: Reconfiguring daemon osd.5 on smithi186 2024-03-22T05:45:41.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:41 smithi005 ceph-mon[89904]: from='client.? 172.21.15.5:0/3136461754' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:45:41.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:41 smithi005 ceph-mon[84705]: from='client.64108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:45:41.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:41 smithi005 ceph-mon[84705]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:45:41.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:41 smithi005 ceph-mon[84705]: from='client.64114 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:45:41.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:41 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:41.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:41 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:41.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:41 smithi005 ceph-mon[84705]: Reconfiguring osd.5 (monmap changed)... 2024-03-22T05:45:41.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:41 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-22T05:45:41.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:41 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:45:41.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:41 smithi005 ceph-mon[84705]: Reconfiguring daemon osd.5 on smithi186 2024-03-22T05:45:41.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:41 smithi005 ceph-mon[84705]: from='client.? 172.21.15.5:0/3136461754' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:45:42.352 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:42 smithi186 ceph-mon[67903]: from='client.54192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:45:42.353 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:42 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:42.353 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:42 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:42.353 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:42 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-22T05:45:42.353 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:42 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:45:42.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:42 smithi005 ceph-mon[84705]: from='client.54192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:45:42.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:42 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:42.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:42 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:42.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:42 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-22T05:45:42.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:42 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:45:42.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:42 smithi005 ceph-mon[89904]: from='client.54192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:45:42.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:42 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:42.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:42 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:42.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:42 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-22T05:45:42.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:42 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:45:43.327 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:43 smithi186 ceph-mon[67903]: Reconfiguring osd.6 (monmap changed)... 2024-03-22T05:45:43.328 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:43 smithi186 ceph-mon[67903]: Reconfiguring daemon osd.6 on smithi186 2024-03-22T05:45:43.328 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:43 smithi186 ceph-mon[67903]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:45:43.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:43 smithi005 ceph-mon[84705]: Reconfiguring osd.6 (monmap changed)... 2024-03-22T05:45:43.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:43 smithi005 ceph-mon[84705]: Reconfiguring daemon osd.6 on smithi186 2024-03-22T05:45:43.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:43 smithi005 ceph-mon[84705]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:45:43.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:43 smithi005 ceph-mon[89904]: Reconfiguring osd.6 (monmap changed)... 2024-03-22T05:45:43.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:43 smithi005 ceph-mon[89904]: Reconfiguring daemon osd.6 on smithi186 2024-03-22T05:45:43.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:43 smithi005 ceph-mon[89904]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:45:44.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:44 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:44.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:44 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:44.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:44 smithi005 ceph-mon[84705]: Reconfiguring osd.7 (monmap changed)... 2024-03-22T05:45:44.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:44 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-22T05:45:44.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:44 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:45:44.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:44 smithi005 ceph-mon[84705]: Reconfiguring daemon osd.7 on smithi186 2024-03-22T05:45:44.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:44 smithi005 ceph-mon[84705]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:45:44.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:44 smithi005 ceph-mon[84705]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:45:44.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:44 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:44.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:44 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:44.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:44 smithi005 ceph-mon[89904]: Reconfiguring osd.7 (monmap changed)... 2024-03-22T05:45:44.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:44 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-22T05:45:44.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:44 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:45:44.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:44 smithi005 ceph-mon[89904]: Reconfiguring daemon osd.7 on smithi186 2024-03-22T05:45:44.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:44 smithi005 ceph-mon[89904]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:45:44.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:44 smithi005 ceph-mon[89904]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:45:44.717 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:44 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:44.717 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:44 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:44.717 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:44 smithi186 ceph-mon[67903]: Reconfiguring osd.7 (monmap changed)... 2024-03-22T05:45:44.717 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:44 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-22T05:45:44.717 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:44 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:45:44.717 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:44 smithi186 ceph-mon[67903]: Reconfiguring daemon osd.7 on smithi186 2024-03-22T05:45:44.717 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:44 smithi186 ceph-mon[67903]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:45:44.717 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:44 smithi186 ceph-mon[67903]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:45:46.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:46 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:46.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:46 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:46.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:46 smithi186 ceph-mon[67903]: Reconfiguring rgw.foo.smithi186.mcgodk (monmap changed)... 2024-03-22T05:45:46.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:46 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi186.mcgodk", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-22T05:45:46.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:46 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi186.mcgodk", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-22T05:45:46.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:46 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:45:46.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:46 smithi186 ceph-mon[67903]: Reconfiguring daemon rgw.foo.smithi186.mcgodk on smithi186 2024-03-22T05:45:46.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:46 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:46.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:46 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:46.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:46 smithi005 ceph-mon[89904]: Reconfiguring rgw.foo.smithi186.mcgodk (monmap changed)... 2024-03-22T05:45:46.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:46 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi186.mcgodk", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-22T05:45:46.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:46 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi186.mcgodk", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-22T05:45:46.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:46 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:45:46.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:46 smithi005 ceph-mon[89904]: Reconfiguring daemon rgw.foo.smithi186.mcgodk on smithi186 2024-03-22T05:45:46.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:46 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:46.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:46 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:46.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:46 smithi005 ceph-mon[84705]: Reconfiguring rgw.foo.smithi186.mcgodk (monmap changed)... 2024-03-22T05:45:46.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:46 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi186.mcgodk", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-22T05:45:46.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:46 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi186.mcgodk", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-22T05:45:46.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:46 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:45:46.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:46 smithi005 ceph-mon[84705]: Reconfiguring daemon rgw.foo.smithi186.mcgodk on smithi186 2024-03-22T05:45:47.452 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:46 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:45:46] "GET /metrics HTTP/1.1" 200 38007 "" "Prometheus/2.43.0" 2024-03-22T05:45:47.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:45:47.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:47.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:47.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:45:47.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:45:47.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:45:47.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:47.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-22T05:45:47.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-22T05:45:47.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-03-22T05:45:47.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-22T05:45:47.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-22T05:45:47.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-03-22T05:45:47.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:45:47.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:47.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:45:47.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:47.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:45:47.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:47.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:45:47.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-22T05:45:47.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-22T05:45:47.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-22T05:45:47.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-22T05:45:47.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:45:47.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:47.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:45:47.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:47.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:45:47.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:47.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:45:47.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:45:47.455 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:47.455 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:47.455 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:45:47.455 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:45:47.455 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:45:47.455 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:47.455 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-22T05:45:47.455 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-22T05:45:47.455 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-03-22T05:45:47.455 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-22T05:45:47.455 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-22T05:45:47.455 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-03-22T05:45:47.455 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:45:47.455 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:47.456 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:45:47.456 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:47.456 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:45:47.456 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:47.456 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:45:47.456 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-22T05:45:47.456 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-22T05:45:47.456 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-22T05:45:47.456 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-22T05:45:47.456 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:45:47.456 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:47.456 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:45:47.456 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:47.456 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:45:47.456 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:47.457 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:45:47.457 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-22T05:45:47.457 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-22T05:45:47.457 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-22T05:45:47.457 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-22T05:45:47.457 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:45:47.457 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-22T05:45:47.457 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-22T05:45:47.457 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-22T05:45:47.458 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-22T05:45:47.458 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:45:47.458 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-22T05:45:47.458 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-22T05:45:47.458 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-22T05:45:47.458 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-22T05:45:47.458 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:45:47.458 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-22T05:45:47.458 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-22T05:45:47.458 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-22T05:45:47.458 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-22T05:45:47.458 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:45:47.458 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-22T05:45:47.458 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-22T05:45:47.459 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-22T05:45:47.459 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-22T05:45:47.459 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:45:47.459 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-22T05:45:47.459 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-22T05:45:47.459 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-22T05:45:47.459 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-22T05:45:47.459 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-22T05:45:47.459 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-22T05:45:47.459 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-22T05:45:47.459 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:45:47.459 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:45:47.459 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-22T05:45:47.459 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-22T05:45:47.459 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-22T05:45:47.460 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-22T05:45:47.460 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-22T05:45:47.460 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-22T05:45:47.460 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-22T05:45:47.460 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-22T05:45:47.460 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-22T05:45:47.460 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-22T05:45:47.460 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-22T05:45:47.460 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-22T05:45:47.460 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-22T05:45:47.460 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-22T05:45:47.460 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-22T05:45:47.461 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-22T05:45:47.461 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-22T05:45:47.461 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-22T05:45:47.461 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:45:47.461 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-22T05:45:47.461 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-22T05:45:47.461 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-22T05:45:47.461 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-22T05:45:47.461 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:45:47.461 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-22T05:45:47.461 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-22T05:45:47.461 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-22T05:45:47.461 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-22T05:45:47.462 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:45:47.462 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-22T05:45:47.462 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-22T05:45:47.462 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-22T05:45:47.462 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-22T05:45:47.462 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:45:47.462 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-22T05:45:47.462 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-22T05:45:47.462 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-22T05:45:47.462 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-22T05:45:47.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:45:47.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:45:47.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-22T05:45:47.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-22T05:45:47.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-22T05:45:47.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-22T05:45:47.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-22T05:45:47.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-22T05:45:47.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-22T05:45:47.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-22T05:45:47.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-22T05:45:47.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-22T05:45:47.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-22T05:45:47.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:45:47.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:45:47.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:45:47.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:45:47.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:45:47.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:45:47.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:45:47.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:45:47.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:45:47.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:45:47.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:45:47.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:45:47.465 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-22T05:45:47.465 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-22T05:45:47.465 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-22T05:45:47.465 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:45:47.465 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:45:47.465 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:45:47.465 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:47.465 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:45:47.465 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:45:47.465 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:45:47.466 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:45:47.466 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-22T05:45:47.466 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-22T05:45:47.466 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-22T05:45:47.466 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-22T05:45:47.466 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-22T05:45:47.466 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-22T05:45:47.466 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-22T05:45:47.466 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:45:47.466 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:45:47.466 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-22T05:45:47.467 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-22T05:45:47.467 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-22T05:45:47.467 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-22T05:45:47.467 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-22T05:45:47.467 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-22T05:45:47.467 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-22T05:45:47.467 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-22T05:45:47.467 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-22T05:45:47.467 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-22T05:45:47.467 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-22T05:45:47.467 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-22T05:45:47.467 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-22T05:45:47.467 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-22T05:45:47.467 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:45:47.468 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:45:47.468 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-22T05:45:47.468 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-22T05:45:47.468 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-22T05:45:47.468 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-22T05:45:47.468 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-22T05:45:47.468 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-22T05:45:47.468 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-22T05:45:47.468 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-22T05:45:47.468 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-22T05:45:47.468 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-22T05:45:47.468 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-22T05:45:47.468 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:45:47.468 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:45:47.468 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:45:47.469 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:45:47.469 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:45:47.469 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:45:47.469 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:45:47.469 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:45:47.469 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:45:47.469 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:45:47.469 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:45:47.469 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:45:47.469 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-22T05:45:47.469 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-22T05:45:47.469 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-22T05:45:47.469 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:45:47.469 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:45:47.470 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:45:47.470 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:47.470 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:45:47.470 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:45:47.470 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:47 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:45:47.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:45:47.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:47.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:47.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:45:47.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:45:47.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:45:47.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:47.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-22T05:45:47.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-22T05:45:47.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-03-22T05:45:47.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-22T05:45:47.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-22T05:45:47.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-03-22T05:45:47.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:45:47.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:47.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:45:47.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:47.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:45:47.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:47.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:45:47.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-22T05:45:47.546 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-22T05:45:47.546 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-22T05:45:47.546 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-22T05:45:47.546 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:45:47.546 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:47.546 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:45:47.546 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:47.546 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:45:47.546 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:47.546 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:45:47.547 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-22T05:45:47.547 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-22T05:45:47.547 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-22T05:45:47.547 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-22T05:45:47.547 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:45:47.547 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-22T05:45:47.547 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-22T05:45:47.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-22T05:45:47.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-22T05:45:47.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:45:47.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-22T05:45:47.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-22T05:45:47.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-22T05:45:47.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-22T05:45:47.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:45:47.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-22T05:45:47.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-22T05:45:47.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-22T05:45:47.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-22T05:45:47.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:45:47.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-22T05:45:47.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-22T05:45:47.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-22T05:45:47.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-22T05:45:47.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:45:47.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-22T05:45:47.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-22T05:45:47.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-22T05:45:47.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-22T05:45:47.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-22T05:45:47.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-22T05:45:47.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-22T05:45:47.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:45:47.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:45:47.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-22T05:45:47.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-22T05:45:47.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-22T05:45:47.550 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-22T05:45:47.550 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-22T05:45:47.550 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-22T05:45:47.550 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-22T05:45:47.550 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-22T05:45:47.550 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-22T05:45:47.550 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-22T05:45:47.550 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-22T05:45:47.550 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-22T05:45:47.550 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-22T05:45:47.550 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-22T05:45:47.550 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:45:47.550 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:45:47.550 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-22T05:45:47.551 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-22T05:45:47.551 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-22T05:45:47.551 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-22T05:45:47.551 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-22T05:45:47.551 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-22T05:45:47.551 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-22T05:45:47.551 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-22T05:45:47.551 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-22T05:45:47.551 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-22T05:45:47.551 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-22T05:45:47.551 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:45:47.551 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:45:47.551 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:45:47.552 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:45:47.552 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:45:47.552 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:45:47.552 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:45:47.552 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:45:47.552 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:45:47.552 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:45:47.552 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:45:47.552 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:45:47.552 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-22T05:45:47.552 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-22T05:45:47.552 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-22T05:45:47.552 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:45:47.553 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:45:47.553 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:45:47.553 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:47.553 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:45:47.553 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:45:47.553 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:47 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:45:48.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-mon[84705]: Upgrade: Setting container_image for all mon 2024-03-22T05:45:48.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-mon[84705]: Upgrade: Setting container_image for all crash 2024-03-22T05:45:48.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-mon[84705]: Upgrade: Setting container_image for all mds 2024-03-22T05:45:48.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-mon[84705]: Upgrade: Setting container_image for all rbd-mirror 2024-03-22T05:45:48.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-mon[84705]: Upgrade: Setting container_image for all cephfs-mirror 2024-03-22T05:45:48.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-mon[84705]: Upgrade: Setting container_image for all ceph-exporter 2024-03-22T05:45:48.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-mon[84705]: Upgrade: Setting container_image for all nfs 2024-03-22T05:45:48.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-mon[84705]: Upgrade: Setting container_image for all nvmeof 2024-03-22T05:45:48.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-mon[84705]: Upgrade: Setting container_image for all node-exporter 2024-03-22T05:45:48.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-mon[84705]: Upgrade: Setting container_image for all prometheus 2024-03-22T05:45:48.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-mon[84705]: Upgrade: Setting container_image for all alertmanager 2024-03-22T05:45:48.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-mon[84705]: Upgrade: Setting container_image for all grafana 2024-03-22T05:45:48.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-mon[84705]: Upgrade: Setting container_image for all loki 2024-03-22T05:45:48.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-mon[84705]: Upgrade: Setting container_image for all promtail 2024-03-22T05:45:48.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-mon[84705]: Upgrade: Finalizing container_image settings 2024-03-22T05:45:48.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-mon[84705]: Upgrade: Complete! 2024-03-22T05:45:48.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-mon[84705]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 925 B/s rd, 0 op/s 2024-03-22T05:45:48.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-mon[84705]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:45:48.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-mon[84705]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:45:48.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-mon[84705]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:45:48.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:48.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-mon[89904]: Upgrade: Setting container_image for all mon 2024-03-22T05:45:48.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-mon[89904]: Upgrade: Setting container_image for all crash 2024-03-22T05:45:48.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-mon[89904]: Upgrade: Setting container_image for all mds 2024-03-22T05:45:48.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-mon[89904]: Upgrade: Setting container_image for all rbd-mirror 2024-03-22T05:45:48.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-mon[89904]: Upgrade: Setting container_image for all cephfs-mirror 2024-03-22T05:45:48.455 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-mon[89904]: Upgrade: Setting container_image for all ceph-exporter 2024-03-22T05:45:48.455 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-mon[89904]: Upgrade: Setting container_image for all nfs 2024-03-22T05:45:48.455 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-mon[89904]: Upgrade: Setting container_image for all nvmeof 2024-03-22T05:45:48.455 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-mon[89904]: Upgrade: Setting container_image for all node-exporter 2024-03-22T05:45:48.455 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-mon[89904]: Upgrade: Setting container_image for all prometheus 2024-03-22T05:45:48.455 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-mon[89904]: Upgrade: Setting container_image for all alertmanager 2024-03-22T05:45:48.455 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-mon[89904]: Upgrade: Setting container_image for all grafana 2024-03-22T05:45:48.455 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-mon[89904]: Upgrade: Setting container_image for all loki 2024-03-22T05:45:48.455 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-mon[89904]: Upgrade: Setting container_image for all promtail 2024-03-22T05:45:48.455 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-mon[89904]: Upgrade: Finalizing container_image settings 2024-03-22T05:45:48.455 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-mon[89904]: Upgrade: Complete! 2024-03-22T05:45:48.455 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-mon[89904]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 925 B/s rd, 0 op/s 2024-03-22T05:45:48.455 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-mon[89904]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:45:48.455 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-mon[89904]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:45:48.455 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-mon[89904]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:45:48.456 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:48.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:48 smithi186 ceph-mon[67903]: Upgrade: Setting container_image for all mon 2024-03-22T05:45:48.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:48 smithi186 ceph-mon[67903]: Upgrade: Setting container_image for all crash 2024-03-22T05:45:48.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:48 smithi186 ceph-mon[67903]: Upgrade: Setting container_image for all mds 2024-03-22T05:45:48.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:48 smithi186 ceph-mon[67903]: Upgrade: Setting container_image for all rbd-mirror 2024-03-22T05:45:48.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:48 smithi186 ceph-mon[67903]: Upgrade: Setting container_image for all cephfs-mirror 2024-03-22T05:45:48.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:48 smithi186 ceph-mon[67903]: Upgrade: Setting container_image for all ceph-exporter 2024-03-22T05:45:48.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:48 smithi186 ceph-mon[67903]: Upgrade: Setting container_image for all nfs 2024-03-22T05:45:48.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:48 smithi186 ceph-mon[67903]: Upgrade: Setting container_image for all nvmeof 2024-03-22T05:45:48.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:48 smithi186 ceph-mon[67903]: Upgrade: Setting container_image for all node-exporter 2024-03-22T05:45:48.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:48 smithi186 ceph-mon[67903]: Upgrade: Setting container_image for all prometheus 2024-03-22T05:45:48.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:48 smithi186 ceph-mon[67903]: Upgrade: Setting container_image for all alertmanager 2024-03-22T05:45:48.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:48 smithi186 ceph-mon[67903]: Upgrade: Setting container_image for all grafana 2024-03-22T05:45:48.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:48 smithi186 ceph-mon[67903]: Upgrade: Setting container_image for all loki 2024-03-22T05:45:48.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:48 smithi186 ceph-mon[67903]: Upgrade: Setting container_image for all promtail 2024-03-22T05:45:48.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:48 smithi186 ceph-mon[67903]: Upgrade: Finalizing container_image settings 2024-03-22T05:45:48.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:48 smithi186 ceph-mon[67903]: Upgrade: Complete! 2024-03-22T05:45:48.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:48 smithi186 ceph-mon[67903]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 925 B/s rd, 0 op/s 2024-03-22T05:45:48.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:48 smithi186 ceph-mon[67903]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:45:48.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:48 smithi186 ceph-mon[67903]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:45:48.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:48 smithi186 ceph-mon[67903]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:45:48.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:48 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:48.952 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:45:48.704+0000 7f7f0ecc9700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:45:48.952 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:45:48.952 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:45:48.952 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:45:48.952 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:45:48.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:45:48.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:45:48.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:45:48.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:45:48.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:45:48.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:45:48.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:45:48.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:45:48.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:45:48.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:45:48.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:45:48.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:45:48.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:45:48.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:45:48.954 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:45:48.954 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:45:48.954 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:45:48.954 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:45:48.954 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:45:48.954 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:45:48.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:45:48.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:45:48.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:45:48.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:45:48.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return future.result(timeout) 2024-03-22T05:45:48.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:45:48.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.__get_result() 2024-03-22T05:45:48.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:45:48.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: raise self._exception 2024-03-22T05:45:48.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:45:48.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: config_blobs=daemon_spec.final_config, 2024-03-22T05:45:48.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:45:48.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:45:48.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:45:48.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:45:48.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:45:48.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:45:48.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:45:48.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:45:48.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:45:48.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:45:48.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:45:48.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:45:48.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:45:48.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:45:48.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:45:48.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:45:48.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:45:48.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:45:48.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:45:48.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:45:48.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:45:48.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:45:48.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:45:49.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:49 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:45:49.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:49 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:45:49.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:49 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:45:49.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:49 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:49.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:49 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:45:49.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:49 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:45:49.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:49 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:45:49.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:49 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:45:49.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:49 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:45:49.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:49 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:45:49.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:49 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:49.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:49 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:45:49.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:49 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:45:49.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:49 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:45:49.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:49 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:45:49.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:49 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:45:49.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:49 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:45:49.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:49 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:45:49.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:49 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:45:49.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:49 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:45:49.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:49 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:45:50.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:50 smithi005 ceph-mon[84705]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:45:50.453 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:45:50.453 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:45:50.453 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:45:50.453 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:45:50.453 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:45:50.453 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:45:50.453 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:45:50.453 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:45:50.453 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:45:50.453 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:45:50.453 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:45:50.453 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:45:50.453 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:45:50.454 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:45:50.454 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:45:50.454 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:45:50.454 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:45:50.454 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:45:50.454 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:45:50.454 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:45:50.454 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:45:50.454 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:45:50.454 INFO:journalctl@ceph.mon.a.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:45:50.454 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:45:50.454 INFO:journalctl@ceph.mon.a.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:45:50.454 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:45:50.454 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:45:50.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:45:50.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: return future.result(timeout) 2024-03-22T05:45:50.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:45:50.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.__get_result() 2024-03-22T05:45:50.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:45:50.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: raise self._exception 2024-03-22T05:45:50.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:45:50.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:45:50.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:45:50.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:45:50.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:45:50.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:45:50.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:45:50.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:45:50.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:45:50.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:45:50.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:45:50.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:45:50.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:45:50.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:45:50.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:45:50.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:45:50.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:45:50.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:45:50.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:45:50.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:45:50.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:45:50.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:45:50.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:45:50.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:45:50.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:45:50.457 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:50 smithi005 ceph-mon[84705]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:45:50.457 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:50 smithi005 ceph-mon[84705]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-22T05:45:50.457 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:50 smithi005 ceph-mon[84705]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:45:50.457 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:50 smithi005 ceph-mon[84705]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:45:50.457 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:50 smithi005 ceph-mon[89904]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:45:50.457 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:45:50.458 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:45:50.458 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:45:50.458 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:45:50.458 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:45:50.458 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:45:50.458 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:45:50.458 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:45:50.458 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:45:50.458 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:45:50.458 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:45:50.458 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:45:50.458 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:45:50.458 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:45:50.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:45:50.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:45:50.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:45:50.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:45:50.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:45:50.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:45:50.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:45:50.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:45:50.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:45:50.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:45:50.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:45:50.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:45:50.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:45:50.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:45:50.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: return future.result(timeout) 2024-03-22T05:45:50.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:45:50.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.__get_result() 2024-03-22T05:45:50.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:45:50.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: raise self._exception 2024-03-22T05:45:50.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:45:50.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:45:50.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:45:50.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:45:50.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:45:50.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:45:50.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:45:50.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:45:50.461 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:45:50.461 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:45:50.461 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:45:50.461 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:45:50.461 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:45:50.461 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:45:50.461 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:45:50.461 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:45:50.461 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:45:50.461 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:45:50.461 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:45:50.461 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:45:50.461 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:45:50.461 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:45:50.461 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:45:50.462 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:45:50.462 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:45:50.462 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:50 smithi005 ceph-mon[89904]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:45:50.462 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:50 smithi005 ceph-mon[89904]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-22T05:45:50.462 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:50 smithi005 ceph-mon[89904]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:45:50.462 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:50 smithi005 ceph-mon[89904]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:45:50.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:50 smithi186 ceph-mon[67903]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:45:50.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:45:50.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:45:50.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:45:50.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:45:50.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:45:50.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:45:50.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:45:50.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:45:50.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:45:50.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:45:50.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:45:50.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:45:50.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:45:50.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:45:50.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:45:50.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:45:50.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:45:50.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:45:50.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:45:50.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:45:50.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:45:50.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:45:50.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:45:50.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:45:50.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:45:50.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:45:50.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:45:50.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:45:50.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: return future.result(timeout) 2024-03-22T05:45:50.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:45:50.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.__get_result() 2024-03-22T05:45:50.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:45:50.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: raise self._exception 2024-03-22T05:45:50.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:45:50.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:45:50.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:45:50.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:45:50.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:45:50.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:45:50.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:45:50.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:45:50.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:45:50.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:45:50.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:45:50.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:45:50.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:45:50.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:45:50.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:45:50.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:45:50.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:45:50.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:45:50.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:45:50.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:45:50.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:45:50.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:45:50.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:45:50.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:45:50.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:45:50.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:50 smithi186 ceph-mon[67903]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:45:50.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:50 smithi186 ceph-mon[67903]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-22T05:45:50.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:50 smithi186 ceph-mon[67903]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:45:50.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:50 smithi186 ceph-mon[67903]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:45:50.952 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:50 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:45:50.673+0000 7f7f0ecc9700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:45:50.952 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:50 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:45:50.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:50 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:45:50.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:50 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:45:50.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:50 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:45:50.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:50 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:45:50.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:50 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:45:50.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:50 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:45:50.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:50 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:45:50.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:50 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:45:50.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:50 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:45:50.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:50 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:45:50.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:50 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:45:50.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:50 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:45:50.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:50 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:45:50.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:50 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:45:50.954 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:50 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:45:50.954 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:50 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:45:50.954 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:50 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:45:50.954 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:50 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:45:50.954 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:50 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:45:50.954 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:50 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:45:50.954 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:50 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:45:50.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:50 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:45:50.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:50 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:45:50.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:50 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:45:50.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:50 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:45:50.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:50 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:45:50.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:50 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:45:50.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:50 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return future.result(timeout) 2024-03-22T05:45:50.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:50 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:45:50.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:50 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.__get_result() 2024-03-22T05:45:50.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:50 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:45:50.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:50 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: raise self._exception 2024-03-22T05:45:50.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:50 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:45:50.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:50 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: config_blobs=daemon_spec.final_config, 2024-03-22T05:45:50.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:50 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:45:50.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:50 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:45:50.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:50 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:45:50.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:50 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:45:50.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:50 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:45:50.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:50 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:45:50.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:50 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:45:50.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:50 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:45:50.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:50 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:45:50.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:50 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:45:50.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:50 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:45:50.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:50 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:45:50.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:50 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:45:50.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:50 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:45:50.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:50 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:45:50.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:50 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:45:50.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:50 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:45:50.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:50 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:45:50.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:50 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:45:50.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:50 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:45:50.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:50 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:45:50.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:50 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:45:50.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:50 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:45:52.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:52 smithi005 ceph-mon[84705]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:45:52.452 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:45:52.452 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:45:52.452 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:45:52.452 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:45:52.453 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:45:52.453 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:45:52.453 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:45:52.453 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:45:52.453 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:45:52.453 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:45:52.453 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:45:52.453 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:45:52.453 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:45:52.453 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:45:52.453 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:45:52.453 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:45:52.453 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:45:52.453 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:45:52.454 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:45:52.454 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:45:52.454 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:45:52.454 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:45:52.454 INFO:journalctl@ceph.mon.a.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:45:52.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:45:52.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:45:52.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:45:52.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:45:52.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:45:52.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: return future.result(timeout) 2024-03-22T05:45:52.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:45:52.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.__get_result() 2024-03-22T05:45:52.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:45:52.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: raise self._exception 2024-03-22T05:45:52.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:45:52.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:45:52.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:45:52.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:45:52.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:45:52.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:45:52.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:45:52.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:45:52.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:45:52.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:45:52.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:45:52.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:45:52.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:45:52.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:45:52.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:45:52.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:45:52.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:45:52.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:45:52.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:45:52.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:45:52.457 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:52 smithi005 ceph-mon[89904]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:45:52.458 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:45:52.458 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:45:52.458 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:45:52.458 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:45:52.458 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:45:52.458 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:45:52.458 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:45:52.458 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:45:52.458 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:45:52.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:45:52.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:45:52.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:45:52.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:45:52.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:45:52.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:45:52.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:45:52.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:45:52.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:45:52.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:45:52.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:45:52.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:45:52.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:45:52.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:45:52.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:45:52.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:45:52.461 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:45:52.461 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:45:52.461 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:45:52.461 INFO:journalctl@ceph.mon.c.smithi005.stdout: return future.result(timeout) 2024-03-22T05:45:52.461 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:45:52.461 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.__get_result() 2024-03-22T05:45:52.461 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:45:52.461 INFO:journalctl@ceph.mon.c.smithi005.stdout: raise self._exception 2024-03-22T05:45:52.461 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:45:52.461 INFO:journalctl@ceph.mon.c.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:45:52.462 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:45:52.462 INFO:journalctl@ceph.mon.c.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:45:52.462 INFO:journalctl@ceph.mon.c.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:45:52.462 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:45:52.462 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:45:52.462 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:45:52.462 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:45:52.462 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:45:52.462 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:45:52.462 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:45:52.462 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:45:52.462 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:45:52.462 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:45:52.462 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:45:52.463 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:45:52.463 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:45:52.463 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:45:52.463 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:45:52.463 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:45:52.463 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:45:52.463 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:45:52.463 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:45:52.463 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:45:52.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:52 smithi005 ceph-mon[84705]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 739 B/s rd, 0 op/s 2024-03-22T05:45:52.464 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:45:52.464 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:45:52.464 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:45:52.464 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:45:52.464 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:45:52.464 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:52 smithi005 ceph-mon[89904]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 739 B/s rd, 0 op/s 2024-03-22T05:45:52.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:52 smithi186 ceph-mon[67903]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:45:52.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:45:52.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:45:52.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:45:52.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:45:52.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:45:52.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:45:52.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:45:52.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:45:52.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:45:52.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:45:52.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:45:52.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:45:52.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:45:52.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:45:52.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:45:52.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:45:52.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:45:52.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:45:52.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:45:52.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:45:52.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:45:52.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:45:52.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:45:52.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:45:52.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:45:52.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:45:52.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:45:52.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:45:52.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: return future.result(timeout) 2024-03-22T05:45:52.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:45:52.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.__get_result() 2024-03-22T05:45:52.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:45:52.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: raise self._exception 2024-03-22T05:45:52.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:45:52.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:45:52.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:45:52.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:45:52.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:45:52.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:45:52.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:45:52.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:45:52.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:45:52.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:45:52.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:45:52.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:45:52.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:45:52.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:45:52.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:45:52.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:45:52.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:45:52.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:45:52.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:45:52.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:45:52.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:45:52.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:45:52.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:45:52.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:45:52.549 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:45:52.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:52 smithi186 ceph-mon[67903]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 739 B/s rd, 0 op/s 2024-03-22T05:45:53.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:53 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:45:53.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:53 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:45:53.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:53 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:45:54.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:54 smithi005 ceph-mon[89904]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 739 B/s rd, 0 op/s 2024-03-22T05:45:54.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:54 smithi005 ceph-mon[84705]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 739 B/s rd, 0 op/s 2024-03-22T05:45:54.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:54 smithi186 ceph-mon[67903]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 739 B/s rd, 0 op/s 2024-03-22T05:45:55.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:55 smithi005 ceph-mon[89904]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:45:55.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:55 smithi005 ceph-mon[84705]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:45:55.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:55 smithi186 ceph-mon[67903]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:45:56.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:56 smithi005 ceph-mon[89904]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-22T05:45:56.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:56 smithi005 ceph-mon[84705]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-22T05:45:56.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:56 smithi186 ceph-mon[67903]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-22T05:45:57.452 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:45:56 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:45:56] "GET /metrics HTTP/1.1" 200 38006 "" "Prometheus/2.43.0" 2024-03-22T05:45:58.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:45:58 smithi005 ceph-mon[89904]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-22T05:45:58.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:45:58 smithi005 ceph-mon[84705]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-22T05:45:58.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:45:58 smithi186 ceph-mon[67903]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-22T05:46:00.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:00 smithi005 ceph-mon[89904]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:46:00.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:00 smithi005 ceph-mon[84705]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:46:00.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:00 smithi186 ceph-mon[67903]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:46:02.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:02 smithi005 ceph-mon[84705]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:46:02.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:02 smithi005 ceph-mon[89904]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:46:02.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:02 smithi186 ceph-mon[67903]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:46:04.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:04 smithi005 ceph-mon[84705]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:46:04.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:04 smithi005 ceph-mon[89904]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:46:04.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:04 smithi186 ceph-mon[67903]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:46:05.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:05 smithi005 ceph-mon[84705]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:46:05.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:05 smithi005 ceph-mon[89904]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:46:05.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:05 smithi186 ceph-mon[67903]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:46:06.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:06 smithi005 ceph-mon[84705]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:46:06.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:06 smithi005 ceph-mon[89904]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:46:06.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:06 smithi186 ceph-mon[67903]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:46:07.452 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:46:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:46:06] "GET /metrics HTTP/1.1" 200 38006 "" "Prometheus/2.43.0" 2024-03-22T05:46:08.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:08 smithi005 ceph-mon[84705]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:46:08.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:08 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:46:08.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:08 smithi005 ceph-mon[89904]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:46:08.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:08 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:46:08.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:08 smithi186 ceph-mon[67903]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:46:08.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:08 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:46:10.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:10 smithi005 ceph-mon[89904]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:46:10.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:10 smithi005 ceph-mon[84705]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:46:10.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:10 smithi186 ceph-mon[67903]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:46:11.817 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph orch ps' 2024-03-22T05:46:12.437 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:12 smithi005 ceph-mon[89904]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:46:12.438 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:12 smithi005 ceph-mon[89904]: from='client.64132 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:46:12.438 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:12 smithi005 ceph-mon[84705]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:46:12.438 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:12 smithi005 ceph-mon[84705]: from='client.64132 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:46:12.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:12 smithi186 ceph-mon[67903]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:46:12.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:12 smithi186 ceph-mon[67903]: from='client.64132 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:46:13.786 INFO:teuthology.orchestra.run.smithi005.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T05:46:13.786 INFO:teuthology.orchestra.run.smithi005.stdout:alertmanager.a smithi005 *:9093,9094 running (3m) 58s ago 25m 19.7M - 0.25.0 c8568f914cd2 ee99dfc435e8 2024-03-22T05:46:13.786 INFO:teuthology.orchestra.run.smithi005.stdout:grafana.a smithi186 *:3000 running (3m) 76s ago 25m 81.5M - 9.4.12 2bacad6d85d8 16fd57dbbd71 2024-03-22T05:46:13.786 INFO:teuthology.orchestra.run.smithi005.stdout:iscsi.foo.smithi005.mcdwbp smithi005 unknown 58s ago 24m 62.4M - 3.5 e1d6a67b021e 5215e03bf81b 2024-03-22T05:46:13.786 INFO:teuthology.orchestra.run.smithi005.stdout:mgr.x smithi186 *:8443,9283,8765 running (4m) 76s ago 30m 448M - 19.0.0-2299-gea01217f aa461e7c5e7f c556421a73f3 2024-03-22T05:46:13.787 INFO:teuthology.orchestra.run.smithi005.stdout:mgr.y smithi005 *:8443,9283,8765 running (14m) 58s ago 31m 496M - 19.0.0-2299-gea01217f aa461e7c5e7f 62a813eebc0d 2024-03-22T05:46:13.787 INFO:teuthology.orchestra.run.smithi005.stdout:mon.a smithi005 running (91s) 58s ago 31m 42.7M 2048M 19.0.0-2299-gea01217f aa461e7c5e7f 111a2aa24dd9 2024-03-22T05:46:13.787 INFO:teuthology.orchestra.run.smithi005.stdout:mon.b smithi186 running (2m) 76s ago 30m 39.1M 2048M 19.0.0-2299-gea01217f aa461e7c5e7f c1798feac15b 2024-03-22T05:46:13.787 INFO:teuthology.orchestra.run.smithi005.stdout:mon.c smithi005 running (63s) 58s ago 30m 20.1M 2048M 19.0.0-2299-gea01217f aa461e7c5e7f 708c72ff74f5 2024-03-22T05:46:13.787 INFO:teuthology.orchestra.run.smithi005.stdout:node-exporter.a smithi005 *:9100 running (4m) 58s ago 26m 8627k - 1.5.0 0da6a335fe13 f635cd72b9d3 2024-03-22T05:46:13.787 INFO:teuthology.orchestra.run.smithi005.stdout:node-exporter.b smithi186 *:9100 running (4m) 76s ago 26m 15.1M - 1.5.0 0da6a335fe13 d7e0b3d20191 2024-03-22T05:46:13.787 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0 smithi005 running (29m) 58s ago 29m 53.9M 2993M 17.2.0 e1d6a67b021e ef5c7053dd37 2024-03-22T05:46:13.787 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1 smithi005 running (29m) 58s ago 29m 56.7M 2993M 17.2.0 e1d6a67b021e a1e7ee1017b6 2024-03-22T05:46:13.787 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2 smithi005 running (29m) 58s ago 29m 49.0M 2993M 17.2.0 e1d6a67b021e 462ea63ad0e5 2024-03-22T05:46:13.787 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3 smithi005 running (28m) 58s ago 28m 57.0M 2993M 17.2.0 e1d6a67b021e b25432a7239b 2024-03-22T05:46:13.787 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4 smithi186 running (28m) 76s ago 28m 56.4M 3248M 17.2.0 e1d6a67b021e cd963671f340 2024-03-22T05:46:13.787 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5 smithi186 running (27m) 76s ago 27m 51.7M 3248M 17.2.0 e1d6a67b021e f98c87f1a2cd 2024-03-22T05:46:13.787 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6 smithi186 running (27m) 76s ago 27m 52.0M 3248M 17.2.0 e1d6a67b021e 704ac49c68c4 2024-03-22T05:46:13.787 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7 smithi186 running (27m) 76s ago 27m 54.3M 3248M 17.2.0 e1d6a67b021e f415b90edf63 2024-03-22T05:46:13.788 INFO:teuthology.orchestra.run.smithi005.stdout:prometheus.a smithi186 *:9095 running (4m) 76s ago 26m 67.2M - 2.43.0 a07b618ecd1d a5a043588d6d 2024-03-22T05:46:13.788 INFO:teuthology.orchestra.run.smithi005.stdout:rgw.foo.smithi005.exdhqu smithi005 *:8000 running (24m) 58s ago 24m 80.5M - 17.2.0 e1d6a67b021e 6bb27fe11553 2024-03-22T05:46:13.788 INFO:teuthology.orchestra.run.smithi005.stdout:rgw.foo.smithi186.mcgodk smithi186 *:8000 running (24m) 76s ago 24m 78.2M - 17.2.0 e1d6a67b021e 19b5c20a34f7 2024-03-22T05:46:14.390 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:14 smithi005 ceph-mon[84705]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:46:14.390 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:14 smithi005 ceph-mon[89904]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:46:14.413 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph versions | jq -e '"'"'.mon | length == 1'"'"'' 2024-03-22T05:46:14.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:14 smithi186 ceph-mon[67903]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:46:15.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:15 smithi005 ceph-mon[84705]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:46:15.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:15 smithi005 ceph-mon[84705]: from='client.54204 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:46:15.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:15 smithi005 ceph-mon[89904]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:46:15.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:15 smithi005 ceph-mon[89904]: from='client.54204 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:46:15.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:15 smithi186 ceph-mon[67903]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:46:15.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:15 smithi186 ceph-mon[67903]: from='client.54204 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:46:16.359 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:16 smithi005 ceph-mon[84705]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:46:16.360 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:16 smithi005 ceph-mon[89904]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:46:16.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:16 smithi186 ceph-mon[67903]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:46:16.720 INFO:teuthology.orchestra.run.smithi005.stdout:true 2024-03-22T05:46:17.004 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:46:16 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:46:16] "GET /metrics HTTP/1.1" 200 38005 "" "Prometheus/2.43.0" 2024-03-22T05:46:17.288 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:17 smithi005 ceph-mon[84705]: from='client.? 172.21.15.5:0/3931169729' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:46:17.288 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:17 smithi005 ceph-mon[89904]: from='client.? 172.21.15.5:0/3931169729' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:46:17.311 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph versions | jq -e '"'"'.mon | keys'"'"' | grep $sha1' 2024-03-22T05:46:17.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:17 smithi186 ceph-mon[67903]: from='client.? 172.21.15.5:0/3931169729' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:46:18.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:18 smithi005 ceph-mon[84705]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:46:18.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:18 smithi005 ceph-mon[89904]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:46:18.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:18 smithi186 ceph-mon[67903]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:46:19.436 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev)" 2024-03-22T05:46:20.083 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '"'"'.up_to_date | length == 5'"'"'' 2024-03-22T05:46:20.391 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:20 smithi005 ceph-mon[89904]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:46:20.391 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:20 smithi005 ceph-mon[89904]: from='client.? 172.21.15.5:0/483773402' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:46:20.392 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:20 smithi005 ceph-mon[84705]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:46:20.392 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:20 smithi005 ceph-mon[84705]: from='client.? 172.21.15.5:0/483773402' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:46:20.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:20 smithi186 ceph-mon[67903]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:46:20.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:20 smithi186 ceph-mon[67903]: from='client.? 172.21.15.5:0/483773402' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:46:22.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:22 smithi005 ceph-mon[84705]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:46:22.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:22 smithi005 ceph-mon[89904]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:46:22.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:22 smithi186 ceph-mon[67903]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:46:23.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:23 smithi005 ceph-mon[84705]: from='client.64156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:46:23.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:23 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:46:23.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:23 smithi005 ceph-mon[89904]: from='client.64156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:46:23.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:23 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:46:23.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:23 smithi186 ceph-mon[67903]: from='client.64156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:46:23.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:23 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:46:24.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:24 smithi005 ceph-mon[89904]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:46:24.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:24 smithi005 ceph-mon[84705]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:46:24.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:24 smithi186 ceph-mon[67903]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:46:25.446 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:25 smithi005 ceph-mon[89904]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:46:25.446 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:25 smithi005 ceph-mon[84705]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:46:25.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:25 smithi186 ceph-mon[67903]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:46:26.372 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:26 smithi005 ceph-mon[89904]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:46:26.372 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:26 smithi005 ceph-mon[84705]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:46:26.457 INFO:teuthology.orchestra.run.smithi005.stdout:true 2024-03-22T05:46:26.656 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:26 smithi186 ceph-mon[67903]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:46:27.042 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:46:26 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:46:26] "GET /metrics HTTP/1.1" 200 38002 "" "Prometheus/2.43.0" 2024-03-22T05:46:27.071 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph orch upgrade status' 2024-03-22T05:46:28.391 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:28 smithi005 ceph-mon[89904]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:46:28.391 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:28 smithi005 ceph-mon[84705]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:46:28.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:28 smithi186 ceph-mon[67903]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:46:29.099 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-03-22T05:46:29.099 INFO:teuthology.orchestra.run.smithi005.stdout: "target_image": null, 2024-03-22T05:46:29.099 INFO:teuthology.orchestra.run.smithi005.stdout: "in_progress": false, 2024-03-22T05:46:29.099 INFO:teuthology.orchestra.run.smithi005.stdout: "which": "", 2024-03-22T05:46:29.100 INFO:teuthology.orchestra.run.smithi005.stdout: "services_complete": [], 2024-03-22T05:46:29.100 INFO:teuthology.orchestra.run.smithi005.stdout: "progress": null, 2024-03-22T05:46:29.100 INFO:teuthology.orchestra.run.smithi005.stdout: "message": "", 2024-03-22T05:46:29.100 INFO:teuthology.orchestra.run.smithi005.stdout: "is_paused": false 2024-03-22T05:46:29.100 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-03-22T05:46:29.715 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph health detail' 2024-03-22T05:46:30.293 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:30 smithi005 ceph-mon[84705]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:46:30.293 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:30 smithi005 ceph-mon[84705]: from='client.54228 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:46:30.294 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:30 smithi005 ceph-mon[89904]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:46:30.294 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:30 smithi005 ceph-mon[89904]: from='client.54228 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:46:30.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:30 smithi186 ceph-mon[67903]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:46:30.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:30 smithi186 ceph-mon[67903]: from='client.54228 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:46:31.865 INFO:teuthology.orchestra.run.smithi005.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-22T05:46:31.866 INFO:teuthology.orchestra.run.smithi005.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-22T05:46:31.866 INFO:teuthology.orchestra.run.smithi005.stdout: daemon iscsi.foo.smithi005.mcdwbp on smithi005 is in unknown state 2024-03-22T05:46:32.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:32 smithi005 ceph-mon[84705]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:46:32.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:32 smithi005 ceph-mon[84705]: from='client.? 172.21.15.5:0/2346288120' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T05:46:32.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:32 smithi005 ceph-mon[89904]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:46:32.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:32 smithi005 ceph-mon[89904]: from='client.? 172.21.15.5:0/2346288120' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T05:46:32.482 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types osd --limit 2' 2024-03-22T05:46:32.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:32 smithi186 ceph-mon[67903]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:46:32.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:32 smithi186 ceph-mon[67903]: from='client.? 172.21.15.5:0/2346288120' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T05:46:34.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:34 smithi005 ceph-mon[84705]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:46:34.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:34 smithi005 ceph-mon[89904]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:46:34.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:34 smithi186 ceph-mon[67903]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:46:35.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:35 smithi186 ceph-mon[67903]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:46:35.569 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:35 smithi005 ceph-mon[84705]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:46:35.569 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:35 smithi005 ceph-mon[89904]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:46:36.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:36 smithi186 ceph-mon[67903]: from='client.54240 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37", "daemon_types": "osd", "limit": 2, "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:46:36.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:36 smithi186 ceph-mon[67903]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:46:36.549 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:36 smithi005 ceph-mon[84705]: from='client.54240 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37", "daemon_types": "osd", "limit": 2, "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:46:36.549 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:36 smithi005 ceph-mon[84705]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:46:36.549 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:36 smithi005 ceph-mon[89904]: from='client.54240 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37", "daemon_types": "osd", "limit": 2, "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:46:36.550 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:36 smithi005 ceph-mon[89904]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:46:36.602 INFO:teuthology.orchestra.run.smithi005.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37 2024-03-22T05:46:37.195 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:46:36 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:46:36] "GET /metrics HTTP/1.1" 200 38002 "" "Prometheus/2.43.0" 2024-03-22T05:46:37.222 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-03-22T05:46:37.815 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:37 smithi005 ceph-mon[84705]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37 2024-03-22T05:46:37.815 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:37 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:46:37.815 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:37 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:46:37.815 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:37 smithi005 ceph-mon[84705]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:46:37.816 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:37 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:46:37.816 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:37 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:46:37.816 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:37 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:46:37.816 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:37 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:46:37.816 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:37 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:46:37.816 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:37 smithi005 ceph-mon[89904]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37 2024-03-22T05:46:37.816 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:37 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:46:37.816 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:37 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:46:37.816 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:37 smithi005 ceph-mon[89904]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:46:37.816 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:37 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:46:37.816 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:37 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:46:37.816 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:37 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:46:37.817 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:37 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:46:37.817 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:37 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:46:38.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:37 smithi186 ceph-mon[67903]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37 2024-03-22T05:46:38.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:37 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:46:38.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:37 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:46:38.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:37 smithi186 ceph-mon[67903]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:46:38.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:37 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:46:38.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:37 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:46:38.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:37 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:46:38.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:37 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:46:38.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:37 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:46:38.890 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:38 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:46:38.891 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:38 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:46:38.891 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:38 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:46:38.891 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:38 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:46:38.891 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:38 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:46:38.891 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:38 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:46:38.892 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:38 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:46:38.892 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:38 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:46:38.893 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:38 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:46:38.893 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:38 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:46:38.893 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:38 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:46:38.893 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:38 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:46:39.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:38 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:46:39.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:38 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:46:39.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:38 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:46:39.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:38 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:46:39.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:38 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:46:39.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:38 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:46:39.511 INFO:teuthology.orchestra.run.smithi005.stdout:true 2024-03-22T05:46:39.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:39 smithi005 ceph-mon[84705]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 880 B/s rd, 0 op/s 2024-03-22T05:46:39.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:39 smithi005 ceph-mon[84705]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 531 B/s rd, 0 op/s 2024-03-22T05:46:39.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:39 smithi005 ceph-mon[84705]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:46:39.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:39 smithi005 ceph-mon[84705]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:46:39.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:39 smithi005 ceph-mon[89904]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 880 B/s rd, 0 op/s 2024-03-22T05:46:39.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:39 smithi005 ceph-mon[89904]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 531 B/s rd, 0 op/s 2024-03-22T05:46:39.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:39 smithi005 ceph-mon[89904]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:46:39.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:39 smithi005 ceph-mon[89904]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:46:40.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:39 smithi186 ceph-mon[67903]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 880 B/s rd, 0 op/s 2024-03-22T05:46:40.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:39 smithi186 ceph-mon[67903]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 531 B/s rd, 0 op/s 2024-03-22T05:46:40.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:39 smithi186 ceph-mon[67903]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:46:40.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:39 smithi186 ceph-mon[67903]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:46:40.192 INFO:teuthology.orchestra.run.smithi005.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T05:46:40.192 INFO:teuthology.orchestra.run.smithi005.stdout:alertmanager.a smithi005 *:9093,9094 running (4m) 85s ago 26m 19.7M - 0.25.0 c8568f914cd2 ee99dfc435e8 2024-03-22T05:46:40.192 INFO:teuthology.orchestra.run.smithi005.stdout:grafana.a smithi186 *:3000 running (3m) 103s ago 25m 81.5M - 9.4.12 2bacad6d85d8 16fd57dbbd71 2024-03-22T05:46:40.192 INFO:teuthology.orchestra.run.smithi005.stdout:iscsi.foo.smithi005.mcdwbp smithi005 unknown 85s ago 25m 62.4M - 3.5 e1d6a67b021e 5215e03bf81b 2024-03-22T05:46:40.193 INFO:teuthology.orchestra.run.smithi005.stdout:mgr.x smithi186 *:8443,9283,8765 running (5m) 103s ago 30m 448M - 19.0.0-2299-gea01217f aa461e7c5e7f c556421a73f3 2024-03-22T05:46:40.193 INFO:teuthology.orchestra.run.smithi005.stdout:mgr.y smithi005 *:8443,9283,8765 running (15m) 85s ago 32m 496M - 19.0.0-2299-gea01217f aa461e7c5e7f 62a813eebc0d 2024-03-22T05:46:40.193 INFO:teuthology.orchestra.run.smithi005.stdout:mon.a smithi005 running (118s) 85s ago 32m 42.7M 2048M 19.0.0-2299-gea01217f aa461e7c5e7f 111a2aa24dd9 2024-03-22T05:46:40.193 INFO:teuthology.orchestra.run.smithi005.stdout:mon.b smithi186 running (2m) 103s ago 30m 39.1M 2048M 19.0.0-2299-gea01217f aa461e7c5e7f c1798feac15b 2024-03-22T05:46:40.193 INFO:teuthology.orchestra.run.smithi005.stdout:mon.c smithi005 running (90s) 85s ago 31m 20.1M 2048M 19.0.0-2299-gea01217f aa461e7c5e7f 708c72ff74f5 2024-03-22T05:46:40.193 INFO:teuthology.orchestra.run.smithi005.stdout:node-exporter.a smithi005 *:9100 running (4m) 85s ago 26m 8627k - 1.5.0 0da6a335fe13 f635cd72b9d3 2024-03-22T05:46:40.193 INFO:teuthology.orchestra.run.smithi005.stdout:node-exporter.b smithi186 *:9100 running (4m) 103s ago 26m 15.1M - 1.5.0 0da6a335fe13 d7e0b3d20191 2024-03-22T05:46:40.193 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0 smithi005 running (30m) 85s ago 30m 53.9M 2993M 17.2.0 e1d6a67b021e ef5c7053dd37 2024-03-22T05:46:40.193 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1 smithi005 running (29m) 85s ago 29m 56.7M 2993M 17.2.0 e1d6a67b021e a1e7ee1017b6 2024-03-22T05:46:40.193 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2 smithi005 running (29m) 85s ago 29m 49.0M 2993M 17.2.0 e1d6a67b021e 462ea63ad0e5 2024-03-22T05:46:40.193 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3 smithi005 running (29m) 85s ago 29m 57.0M 2993M 17.2.0 e1d6a67b021e b25432a7239b 2024-03-22T05:46:40.193 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4 smithi186 running (28m) 103s ago 28m 56.4M 3248M 17.2.0 e1d6a67b021e cd963671f340 2024-03-22T05:46:40.193 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5 smithi186 running (28m) 103s ago 28m 51.7M 3248M 17.2.0 e1d6a67b021e f98c87f1a2cd 2024-03-22T05:46:40.193 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6 smithi186 running (27m) 103s ago 27m 52.0M 3248M 17.2.0 e1d6a67b021e 704ac49c68c4 2024-03-22T05:46:40.193 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7 smithi186 running (27m) 103s ago 27m 54.3M 3248M 17.2.0 e1d6a67b021e f415b90edf63 2024-03-22T05:46:40.194 INFO:teuthology.orchestra.run.smithi005.stdout:prometheus.a smithi186 *:9095 running (4m) 103s ago 26m 67.2M - 2.43.0 a07b618ecd1d a5a043588d6d 2024-03-22T05:46:40.194 INFO:teuthology.orchestra.run.smithi005.stdout:rgw.foo.smithi005.exdhqu smithi005 *:8000 running (25m) 85s ago 25m 80.5M - 17.2.0 e1d6a67b021e 6bb27fe11553 2024-03-22T05:46:40.194 INFO:teuthology.orchestra.run.smithi005.stdout:rgw.foo.smithi186.mcgodk smithi186 *:8000 running (25m) 103s ago 25m 78.2M - 17.2.0 e1d6a67b021e 19b5c20a34f7 2024-03-22T05:46:40.621 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-03-22T05:46:40.622 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": { 2024-03-22T05:46:40.622 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev)": 3 2024-03-22T05:46:40.622 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:46:40.622 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr": { 2024-03-22T05:46:40.622 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev)": 2 2024-03-22T05:46:40.622 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:46:40.622 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": { 2024-03-22T05:46:40.622 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-22T05:46:40.622 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:46:40.622 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw": { 2024-03-22T05:46:40.622 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-22T05:46:40.623 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:46:40.623 INFO:teuthology.orchestra.run.smithi005.stdout: "overall": { 2024-03-22T05:46:40.623 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 10, 2024-03-22T05:46:40.623 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev)": 5 2024-03-22T05:46:40.623 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-03-22T05:46:40.623 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-03-22T05:46:40.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:40 smithi005 ceph-mon[84705]: from='client.64180 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:46:40.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:40 smithi005 ceph-mon[84705]: from='client.54252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:46:40.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:40 smithi005 ceph-mon[84705]: from='client.54258 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:46:40.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:40 smithi005 ceph-mon[89904]: from='client.64180 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:46:40.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:40 smithi005 ceph-mon[89904]: from='client.54252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:46:40.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:40 smithi005 ceph-mon[89904]: from='client.54258 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:46:40.954 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:46:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:46:40.580+0000 7f7f0ecc9700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:46:40.954 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:46:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:46:40.954 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:46:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:46:40.954 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:46:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:46:40.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:46:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:46:40.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:46:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:46:40.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:46:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:46:40.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:46:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:46:40.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:46:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:46:40.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:46:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:46:40.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:46:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:46:40.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:46:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:46:40.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:46:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:46:40.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:46:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:46:40.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:46:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:46:40.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:46:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:46:40.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:46:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:46:40.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:46:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:46:40.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:46:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:46:40.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:46:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:46:40.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:46:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:46:40.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:46:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:46:40.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:46:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:46:40.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:46:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:46:40.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:46:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:46:40.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:46:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:46:40.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:46:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:46:40.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:46:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:46:40.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:46:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:46:40.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:46:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return future.result(timeout) 2024-03-22T05:46:40.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:46:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:46:40.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:46:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.__get_result() 2024-03-22T05:46:40.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:46:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:46:40.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:46:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: raise self._exception 2024-03-22T05:46:40.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:46:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:46:40.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:46:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: config_blobs=daemon_spec.final_config, 2024-03-22T05:46:40.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:46:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:46:40.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:46:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:46:40.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:46:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:46:40.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:46:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:46:40.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:46:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:46:40.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:46:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:46:40.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:46:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:46:40.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:46:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:46:40.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:46:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:46:40.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:46:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:46:40.958 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:46:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:46:40.958 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:46:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:46:40.958 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:46:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:46:40.958 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:46:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:46:40.958 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:46:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:46:40.958 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:46:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:46:40.958 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:46:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:46:40.958 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:46:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:46:40.958 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:46:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:46:40.958 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:46:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:46:40.958 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:46:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:46:40.958 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:46:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:46:40.958 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:46:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:46:40.973 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-03-22T05:46:40.973 INFO:teuthology.orchestra.run.smithi005.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37", 2024-03-22T05:46:40.973 INFO:teuthology.orchestra.run.smithi005.stdout: "in_progress": true, 2024-03-22T05:46:40.973 INFO:teuthology.orchestra.run.smithi005.stdout: "which": "Upgrading daemons of type(s) osd. Upgrade limited to 2 daemons (2 remaining).", 2024-03-22T05:46:40.973 INFO:teuthology.orchestra.run.smithi005.stdout: "services_complete": [], 2024-03-22T05:46:40.973 INFO:teuthology.orchestra.run.smithi005.stdout: "progress": "", 2024-03-22T05:46:40.974 INFO:teuthology.orchestra.run.smithi005.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37 image", 2024-03-22T05:46:40.974 INFO:teuthology.orchestra.run.smithi005.stdout: "is_paused": false 2024-03-22T05:46:40.974 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-03-22T05:46:41.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:40 smithi186 ceph-mon[67903]: from='client.64180 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:46:41.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:40 smithi186 ceph-mon[67903]: from='client.54252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:46:41.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:40 smithi186 ceph-mon[67903]: from='client.54258 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:46:41.706 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:41 smithi005 ceph-mon[84705]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-22T05:46:41.706 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:41 smithi005 ceph-mon[84705]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:46:41.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:46:41.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:46:41.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:46:41.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:46:41.707 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:46:41.707 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:46:41.707 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:46:41.707 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:46:41.707 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:46:41.707 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:46:41.707 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:46:41.707 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:46:41.707 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:46:41.707 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:46:41.707 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:46:41.707 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:46:41.708 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:46:41.708 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:46:41.708 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:46:41.708 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:46:41.708 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:46:41.708 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:46:41.708 INFO:journalctl@ceph.mon.a.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:46:41.708 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:46:41.708 INFO:journalctl@ceph.mon.a.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:46:41.708 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:46:41.708 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:46:41.708 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:46:41.708 INFO:journalctl@ceph.mon.a.smithi005.stdout: return future.result(timeout) 2024-03-22T05:46:41.709 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:46:41.709 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.__get_result() 2024-03-22T05:46:41.709 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:46:41.709 INFO:journalctl@ceph.mon.a.smithi005.stdout: raise self._exception 2024-03-22T05:46:41.709 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:46:41.709 INFO:journalctl@ceph.mon.a.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:46:41.709 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:46:41.709 INFO:journalctl@ceph.mon.a.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:46:41.709 INFO:journalctl@ceph.mon.a.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:46:41.709 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:46:41.709 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:46:41.709 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:46:41.709 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:46:41.710 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:46:41.710 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:46:41.710 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:46:41.710 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:46:41.710 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:46:41.710 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:46:41.710 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:46:41.710 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:46:41.710 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:46:41.710 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:46:41.710 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:46:41.711 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:41 smithi005 ceph-mon[89904]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-22T05:46:41.711 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:41 smithi005 ceph-mon[89904]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:46:41.711 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:46:41.711 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:46:41.711 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:46:41.711 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:46:41.711 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:46:41.711 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:46:41.712 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:46:41.712 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:46:41.712 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:46:41.712 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:46:41.712 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:46:41.712 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:46:41.712 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:46:41.712 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:46:41.712 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:46:41.712 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:46:41.712 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:46:41.712 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:46:41.713 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:46:41.713 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:46:41.713 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:46:41.713 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:46:41.713 INFO:journalctl@ceph.mon.c.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:46:41.713 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:46:41.713 INFO:journalctl@ceph.mon.c.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:46:41.713 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:46:41.713 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:46:41.713 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:46:41.713 INFO:journalctl@ceph.mon.c.smithi005.stdout: return future.result(timeout) 2024-03-22T05:46:41.713 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:46:41.713 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.__get_result() 2024-03-22T05:46:41.714 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:46:41.714 INFO:journalctl@ceph.mon.c.smithi005.stdout: raise self._exception 2024-03-22T05:46:41.714 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:46:41.714 INFO:journalctl@ceph.mon.c.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:46:41.714 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:46:41.714 INFO:journalctl@ceph.mon.c.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:46:41.714 INFO:journalctl@ceph.mon.c.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:46:41.714 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:46:41.714 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:46:41.714 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:46:41.714 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:46:41.714 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:46:41.715 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:46:41.715 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:46:41.715 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:46:41.715 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:46:41.715 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:46:41.715 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:46:41.715 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:46:41.715 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:46:41.715 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:46:41.715 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:46:41.716 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:46:41.716 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:46:41.716 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:46:41.716 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:46:41.716 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:46:41.716 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:41 smithi005 ceph-mon[84705]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37 2024-03-22T05:46:41.716 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:41 smithi005 ceph-mon[84705]: from='client.? 172.21.15.5:0/2139550324' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:46:41.716 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:41 smithi005 ceph-mon[84705]: from='client.54270 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:46:41.716 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:46:41.717 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:46:41.717 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:46:41.717 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:46:41.717 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:46:41.717 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:41 smithi005 ceph-mon[89904]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37 2024-03-22T05:46:41.717 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:41 smithi005 ceph-mon[89904]: from='client.? 172.21.15.5:0/2139550324' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:46:41.717 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:41 smithi005 ceph-mon[89904]: from='client.54270 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:46:42.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:41 smithi186 ceph-mon[67903]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-22T05:46:42.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:41 smithi186 ceph-mon[67903]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:46:42.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:46:42.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:46:42.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:46:42.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:46:42.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:46:42.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:46:42.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:46:42.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:46:42.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:46:42.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:46:42.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:46:42.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:46:42.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:46:42.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:46:42.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:46:42.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:46:42.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:46:42.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:46:42.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:46:42.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:46:42.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:46:42.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:46:42.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:46:42.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:46:42.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:46:42.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:46:42.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:46:42.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:46:42.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: return future.result(timeout) 2024-03-22T05:46:42.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:46:42.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.__get_result() 2024-03-22T05:46:42.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:46:42.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: raise self._exception 2024-03-22T05:46:42.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:46:42.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:46:42.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:46:42.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:46:42.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:46:42.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:46:42.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:46:42.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:46:42.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:46:42.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:46:42.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:46:42.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:46:42.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:46:42.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:46:42.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:46:42.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:46:42.048 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:46:42.048 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:46:42.048 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:46:42.048 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:46:42.048 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:46:42.048 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:46:42.048 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:46:42.048 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:46:42.048 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:46:42.048 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:41 smithi186 ceph-mon[67903]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37 2024-03-22T05:46:42.049 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:41 smithi186 ceph-mon[67903]: from='client.? 172.21.15.5:0/2139550324' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:46:42.049 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:41 smithi186 ceph-mon[67903]: from='client.54270 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:46:43.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:43 smithi005 ceph-mon[89904]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-22T05:46:43.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:43 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:46:43.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:43 smithi005 ceph-mon[89904]: Upgrade: Target is version 19.0.0-2299-gea01217f (squid) 2024-03-22T05:46:43.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:43 smithi005 ceph-mon[89904]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, digests ['quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9'] 2024-03-22T05:46:43.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:43 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:46:43.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:43 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:46:43.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:43 smithi005 ceph-mon[89904]: Upgrade: Setting container_image for all mgr 2024-03-22T05:46:43.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:43 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:46:43.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:43 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:46:43.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:43 smithi005 ceph-mon[89904]: Upgrade: Setting container_image for all mon 2024-03-22T05:46:43.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:43 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:46:43.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:43 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:46:43.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:43 smithi005 ceph-mon[89904]: Upgrade: Setting container_image for all crash 2024-03-22T05:46:43.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:43 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:46:43.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:43 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-22T05:46:43.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:43 smithi005 ceph-mon[89904]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-22T05:46:43.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:43 smithi005 ceph-mon[89904]: Upgrade: osd.0 is safe to restart 2024-03-22T05:46:43.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:43 smithi005 ceph-mon[84705]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-22T05:46:43.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:43 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:46:43.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:43 smithi005 ceph-mon[84705]: Upgrade: Target is version 19.0.0-2299-gea01217f (squid) 2024-03-22T05:46:43.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:43 smithi005 ceph-mon[84705]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, digests ['quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9'] 2024-03-22T05:46:43.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:43 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:46:43.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:43 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:46:43.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:43 smithi005 ceph-mon[84705]: Upgrade: Setting container_image for all mgr 2024-03-22T05:46:43.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:43 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:46:43.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:43 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:46:43.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:43 smithi005 ceph-mon[84705]: Upgrade: Setting container_image for all mon 2024-03-22T05:46:43.955 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:43 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:46:43.955 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:43 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:46:43.955 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:43 smithi005 ceph-mon[84705]: Upgrade: Setting container_image for all crash 2024-03-22T05:46:43.955 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:43 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:46:43.955 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:43 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-22T05:46:43.955 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:43 smithi005 ceph-mon[84705]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-22T05:46:43.955 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:43 smithi005 ceph-mon[84705]: Upgrade: osd.0 is safe to restart 2024-03-22T05:46:44.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:43 smithi186 ceph-mon[67903]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-22T05:46:44.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:43 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:46:44.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:43 smithi186 ceph-mon[67903]: Upgrade: Target is version 19.0.0-2299-gea01217f (squid) 2024-03-22T05:46:44.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:43 smithi186 ceph-mon[67903]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, digests ['quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9'] 2024-03-22T05:46:44.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:43 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:46:44.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:43 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:46:44.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:43 smithi186 ceph-mon[67903]: Upgrade: Setting container_image for all mgr 2024-03-22T05:46:44.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:43 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:46:44.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:43 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:46:44.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:43 smithi186 ceph-mon[67903]: Upgrade: Setting container_image for all mon 2024-03-22T05:46:44.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:43 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:46:44.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:43 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:46:44.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:43 smithi186 ceph-mon[67903]: Upgrade: Setting container_image for all crash 2024-03-22T05:46:44.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:43 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:46:44.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:43 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-22T05:46:44.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:43 smithi186 ceph-mon[67903]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-22T05:46:44.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:43 smithi186 ceph-mon[67903]: Upgrade: osd.0 is safe to restart 2024-03-22T05:46:45.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:44 smithi186 ceph-mon[67903]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:46:45.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:44 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:46:45.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:44 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-22T05:46:45.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:44 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:46:45.056 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:44 smithi005 ceph-mon[89904]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:46:45.056 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:44 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:46:45.056 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:44 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-22T05:46:45.056 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:44 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:46:45.056 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:44 smithi005 ceph-mon[84705]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:46:45.056 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:44 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:46:45.056 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:44 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-22T05:46:45.056 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:44 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:46:45.897 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:45 smithi005 ceph-mon[89904]: Upgrade: Updating osd.0 2024-03-22T05:46:45.897 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:45 smithi005 ceph-mon[89904]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-22T05:46:45.897 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:45 smithi005 ceph-mon[89904]: Deploying daemon osd.0 on smithi005 2024-03-22T05:46:45.898 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:45 smithi005 ceph-mon[84705]: Upgrade: Updating osd.0 2024-03-22T05:46:45.898 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:45 smithi005 ceph-mon[84705]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-22T05:46:45.898 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:45 smithi005 ceph-mon[84705]: Deploying daemon osd.0 on smithi005 2024-03-22T05:46:46.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:45 smithi186 ceph-mon[67903]: Upgrade: Updating osd.0 2024-03-22T05:46:46.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:45 smithi186 ceph-mon[67903]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-22T05:46:46.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:45 smithi186 ceph-mon[67903]: Deploying daemon osd.0 on smithi005 2024-03-22T05:46:47.338 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:46:46 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:46:46] "GET /metrics HTTP/1.1" 200 38005 "" "Prometheus/2.43.0" 2024-03-22T05:46:47.943 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 22 05:46:47 smithi005 systemd[1]: Stopping Ceph osd.0 for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:46:48.202 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 22 05:46:47 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-0[32361]: 2024-03-22T05:46:47.943+0000 7f129404a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T05:46:48.202 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 22 05:46:47 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-0[32361]: 2024-03-22T05:46:47.943+0000 7f129404a700 -1 osd.0 105 *** Got signal Terminated *** 2024-03-22T05:46:48.202 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 22 05:46:47 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-0[32361]: 2024-03-22T05:46:47.943+0000 7f129404a700 -1 osd.0 105 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-22T05:46:48.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:48 smithi186 ceph-mon[67903]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-22T05:46:48.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:48 smithi186 ceph-mon[67903]: osd.0 marked itself down and dead 2024-03-22T05:46:48.671 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:48 smithi005 ceph-mon[84705]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-22T05:46:48.671 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:48 smithi005 ceph-mon[84705]: osd.0 marked itself down and dead 2024-03-22T05:46:48.672 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:48 smithi005 ceph-mon[89904]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-22T05:46:48.672 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:48 smithi005 ceph-mon[89904]: osd.0 marked itself down and dead 2024-03-22T05:46:48.927 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 22 05:46:48 smithi005 podman[97507]: 2024-03-22 05:46:48.672287032 +0000 UTC m=+0.827109343 container died ef5c7053dd3713407b1f6b8ad3df7f7a03800c692236469c3e86812e6619c127 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, ceph=True, name=centos-stream, GIT_BRANCH=HEAD, version=8, io.buildah.version=1.19.8, io.openshift.expose-services=, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., GIT_CLEAN=True, release=754, distribution-scope=public, architecture=x86_64, io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux ) 2024-03-22T05:46:49.202 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 22 05:46:49 smithi005 podman[97507]: 2024-03-22 05:46:49.018590546 +0000 UTC m=+1.173412862 container cleanup ef5c7053dd3713407b1f6b8ad3df7f7a03800c692236469c3e86812e6619c127 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-0, GIT_CLEAN=True, ceph=True, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, release=754, RELEASE=HEAD, io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, version=8, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux ) 2024-03-22T05:46:49.202 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 22 05:46:49 smithi005 bash[97507]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-0 2024-03-22T05:46:49.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:49 smithi186 ceph-mon[67903]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-22T05:46:49.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:49 smithi186 ceph-mon[67903]: osdmap e106: 8 total, 7 up, 8 in 2024-03-22T05:46:49.639 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 22 05:46:49 smithi005 podman[97525]: 2024-03-22 05:46:49.228460026 +0000 UTC m=+0.554387327 container remove ef5c7053dd3713407b1f6b8ad3df7f7a03800c692236469c3e86812e6619c127 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-0, build-date=2022-05-03T08:36:31.336870, name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, architecture=x86_64, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, distribution-scope=public, com.redhat.component=centos-stream-container, vcs-type=git, RELEASE=HEAD, vendor=Red Hat, Inc., GIT_CLEAN=True) 2024-03-22T05:46:49.639 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:49 smithi005 ceph-mon[84705]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-22T05:46:49.640 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:49 smithi005 ceph-mon[84705]: osdmap e106: 8 total, 7 up, 8 in 2024-03-22T05:46:49.640 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:49 smithi005 ceph-mon[89904]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-22T05:46:49.641 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:49 smithi005 ceph-mon[89904]: osdmap e106: 8 total, 7 up, 8 in 2024-03-22T05:46:49.914 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 22 05:46:49 smithi005 podman[97619]: 2024-03-22 05:46:49.542922172 +0000 UTC m=+0.018133103 image pull aa461e7c5e7f322b169c789b389d594b994ea7ffcbc9e24e563a3ec9b69bc3c1 quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9 2024-03-22T05:46:49.914 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 22 05:46:49 smithi005 podman[97619]: 2024-03-22 05:46:49.645760966 +0000 UTC m=+0.120971888 container create 9aaa6be092ebb11710967a3bf0899d7dcfa0e178a2d5766697805af9b31bdbb8 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-0-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-22T05:46:50.202 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 22 05:46:49 smithi005 podman[97619]: 2024-03-22 05:46:49.915254692 +0000 UTC m=+0.390465622 container init 9aaa6be092ebb11710967a3bf0899d7dcfa0e178a2d5766697805af9b31bdbb8 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-0-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, ceph=True, CEPH_POINT_RELEASE=) 2024-03-22T05:46:50.202 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 22 05:46:49 smithi005 podman[97619]: 2024-03-22 05:46:49.918605187 +0000 UTC m=+0.393816112 container start 9aaa6be092ebb11710967a3bf0899d7dcfa0e178a2d5766697805af9b31bdbb8 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-0-deactivate, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-22T05:46:50.202 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 22 05:46:49 smithi005 podman[97619]: 2024-03-22 05:46:49.998665888 +0000 UTC m=+0.473876810 container attach 9aaa6be092ebb11710967a3bf0899d7dcfa0e178a2d5766697805af9b31bdbb8 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-0-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True) 2024-03-22T05:46:50.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:50 smithi186 ceph-mon[67903]: pgmap v68: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:46:50.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:50 smithi186 ceph-mon[67903]: osdmap e107: 8 total, 7 up, 8 in 2024-03-22T05:46:50.668 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 22 05:46:50 smithi005 podman[97619]: 2024-03-22 05:46:50.209457516 +0000 UTC m=+0.684668457 container died 9aaa6be092ebb11710967a3bf0899d7dcfa0e178a2d5766697805af9b31bdbb8 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-0-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-22T05:46:50.668 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:50 smithi005 ceph-mon[84705]: pgmap v68: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:46:50.668 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:50 smithi005 ceph-mon[84705]: osdmap e107: 8 total, 7 up, 8 in 2024-03-22T05:46:50.669 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:50 smithi005 ceph-mon[89904]: pgmap v68: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:46:50.669 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:50 smithi005 ceph-mon[89904]: osdmap e107: 8 total, 7 up, 8 in 2024-03-22T05:46:50.952 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 22 05:46:50 smithi005 podman[97645]: 2024-03-22 05:46:50.669076323 +0000 UTC m=+0.454437916 container remove 9aaa6be092ebb11710967a3bf0899d7dcfa0e178a2d5766697805af9b31bdbb8 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2) 2024-03-22T05:46:50.952 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 22 05:46:50 smithi005 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@osd.0.service: Deactivated successfully. 2024-03-22T05:46:50.952 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 22 05:46:50 smithi005 systemd[1]: Stopped Ceph osd.0 for f59f36e2-e80a-11ee-95cd-87774f69a715. 2024-03-22T05:46:50.952 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 22 05:46:50 smithi005 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@osd.0.service: Consumed 14.747s CPU time. 2024-03-22T05:46:50.952 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 22 05:46:50 smithi005 systemd[1]: Starting Ceph osd.0 for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:46:51.371 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:51 smithi005 ceph-mon[84705]: Health check failed: Reduced data availability: 12 pgs inactive, 56 pgs peering (PG_AVAILABILITY) 2024-03-22T05:46:51.372 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:51 smithi005 ceph-mon[89904]: Health check failed: Reduced data availability: 12 pgs inactive, 56 pgs peering (PG_AVAILABILITY) 2024-03-22T05:46:51.372 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 22 05:46:51 smithi005 podman[97764]: 2024-03-22 05:46:51.022423357 +0000 UTC m=+0.017874089 image pull aa461e7c5e7f322b169c789b389d594b994ea7ffcbc9e24e563a3ec9b69bc3c1 quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9 2024-03-22T05:46:51.372 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 22 05:46:51 smithi005 podman[97764]: 2024-03-22 05:46:51.136666047 +0000 UTC m=+0.132116759 container create 6eb1587b30ac4eaca20f85c8e6a259f32c81603d76a02e33c4c9c47e95ed2764 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-0-activate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-22T05:46:51.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:51 smithi186 ceph-mon[67903]: Health check failed: Reduced data availability: 12 pgs inactive, 56 pgs peering (PG_AVAILABILITY) 2024-03-22T05:46:51.702 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 22 05:46:51 smithi005 podman[97764]: 2024-03-22 05:46:51.372655917 +0000 UTC m=+0.368106643 container init 6eb1587b30ac4eaca20f85c8e6a259f32c81603d76a02e33c4c9c47e95ed2764 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-0-activate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux ) 2024-03-22T05:46:51.702 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 22 05:46:51 smithi005 podman[97764]: 2024-03-22 05:46:51.375865184 +0000 UTC m=+0.371315898 container start 6eb1587b30ac4eaca20f85c8e6a259f32c81603d76a02e33c4c9c47e95ed2764 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-0-activate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-22T05:46:51.702 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 22 05:46:51 smithi005 podman[97764]: 2024-03-22 05:46:51.589376093 +0000 UTC m=+0.584826808 container attach 6eb1587b30ac4eaca20f85c8e6a259f32c81603d76a02e33c4c9c47e95ed2764 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-0-activate, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True) 2024-03-22T05:46:52.192 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 22 05:46:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-0-activate[97779]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-22T05:46:52.192 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 22 05:46:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-0-activate[97779]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-22T05:46:52.192 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 22 05:46:51 smithi005 bash[97764]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-22T05:46:52.192 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 22 05:46:51 smithi005 bash[97764]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-22T05:46:52.192 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 22 05:46:51 smithi005 bash[97764]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-22T05:46:52.192 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 22 05:46:51 smithi005 bash[97764]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-22T05:46:52.192 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 22 05:46:51 smithi005 bash[97764]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-03-22T05:46:52.192 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 22 05:46:51 smithi005 bash[97764]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-22T05:46:52.192 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 22 05:46:51 smithi005 bash[97764]: --> ceph-volume raw activate successful for osd ID: 0 2024-03-22T05:46:52.192 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 22 05:46:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-0-activate[97779]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-22T05:46:52.193 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 22 05:46:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-0-activate[97779]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-22T05:46:52.193 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 22 05:46:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-0-activate[97779]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-03-22T05:46:52.193 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 22 05:46:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-0-activate[97779]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-22T05:46:52.193 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 22 05:46:51 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-0-activate[97779]: --> ceph-volume raw activate successful for osd ID: 0 2024-03-22T05:46:52.193 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 22 05:46:51 smithi005 podman[97764]: 2024-03-22 05:46:51.909477989 +0000 UTC m=+0.904928703 container died 6eb1587b30ac4eaca20f85c8e6a259f32c81603d76a02e33c4c9c47e95ed2764 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-0-activate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-22T05:46:52.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:52 smithi005 ceph-mon[84705]: pgmap v70: 161 pgs: 58 peering, 103 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:46:52.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:52 smithi005 ceph-mon[89904]: pgmap v70: 161 pgs: 58 peering, 103 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:46:52.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:52 smithi186 ceph-mon[67903]: pgmap v70: 161 pgs: 58 peering, 103 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:46:52.801 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 22 05:46:52 smithi005 podman[97898]: 2024-03-22 05:46:52.459891772 +0000 UTC m=+0.545549563 container remove 6eb1587b30ac4eaca20f85c8e6a259f32c81603d76a02e33c4c9c47e95ed2764 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-0-activate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-22T05:46:52.801 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 22 05:46:52 smithi005 podman[97964]: 2024-03-22 05:46:52.677671494 +0000 UTC m=+0.114367563 container create 31b6cdee5f5b9f92efc9cf537c9ee43b9bdbbc3345f06de68023d7618d3b402c (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-22T05:46:52.802 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 22 05:46:52 smithi005 podman[97964]: 2024-03-22 05:46:52.581919402 +0000 UTC m=+0.018615477 image pull aa461e7c5e7f322b169c789b389d594b994ea7ffcbc9e24e563a3ec9b69bc3c1 quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9 2024-03-22T05:46:53.202 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 22 05:46:52 smithi005 podman[97964]: 2024-03-22 05:46:52.930288184 +0000 UTC m=+0.366984256 container init 31b6cdee5f5b9f92efc9cf537c9ee43b9bdbbc3345f06de68023d7618d3b402c (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-0, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2) 2024-03-22T05:46:53.202 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 22 05:46:52 smithi005 podman[97964]: 2024-03-22 05:46:52.933926977 +0000 UTC m=+0.370623047 container start 31b6cdee5f5b9f92efc9cf537c9ee43b9bdbbc3345f06de68023d7618d3b402c (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-22T05:46:53.202 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 22 05:46:52 smithi005 ceph-osd[97982]: -- 172.21.15.5:0/1572966932 <== mon.0 v2:172.21.15.5:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55696a917680 con 0x55696a8e1400 2024-03-22T05:46:53.202 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 22 05:46:53 smithi005 bash[97964]: 31b6cdee5f5b9f92efc9cf537c9ee43b9bdbbc3345f06de68023d7618d3b402c 2024-03-22T05:46:53.203 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 22 05:46:53 smithi005 systemd[1]: Started Ceph osd.0 for f59f36e2-e80a-11ee-95cd-87774f69a715. 2024-03-22T05:46:53.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:53 smithi186 ceph-mon[67903]: pgmap v71: 161 pgs: 58 peering, 103 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:46:53.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:53 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:46:53.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:53 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:46:53.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:53 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:46:53.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:53 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:46:53.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:53 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:46:53.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:53 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:46:53.901 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 22 05:46:53 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-0[97978]: 2024-03-22T05:46:53.477+0000 7f832c70d7c0 -1 Falling back to public interface 2024-03-22T05:46:53.901 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:53 smithi005 ceph-mon[84705]: pgmap v71: 161 pgs: 58 peering, 103 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:46:53.902 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:53 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:46:53.902 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:53 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:46:53.902 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:53 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:46:53.902 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:53 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:46:53.902 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:53 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:46:53.902 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:53 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:46:53.902 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:53 smithi005 ceph-mon[89904]: pgmap v71: 161 pgs: 58 peering, 103 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:46:53.902 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:53 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:46:53.903 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:53 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:46:53.903 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:53 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:46:53.903 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:53 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:46:53.903 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:53 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:46:53.903 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:53 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:46:54.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:54 smithi186 ceph-mon[67903]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:46:54.808 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:54 smithi005 ceph-mon[84705]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:46:54.809 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:54 smithi005 ceph-mon[89904]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:46:55.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:55 smithi005 ceph-mon[84705]: pgmap v72: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 78/723 objects degraded (10.788%) 2024-03-22T05:46:55.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:55 smithi005 ceph-mon[84705]: Health check failed: Degraded data redundancy: 78/723 objects degraded (10.788%), 22 pgs degraded (PG_DEGRADED) 2024-03-22T05:46:55.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:55 smithi005 ceph-mon[84705]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 12 pgs inactive, 56 pgs peering) 2024-03-22T05:46:55.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:55 smithi005 ceph-mon[89904]: pgmap v72: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 78/723 objects degraded (10.788%) 2024-03-22T05:46:55.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:55 smithi005 ceph-mon[89904]: Health check failed: Degraded data redundancy: 78/723 objects degraded (10.788%), 22 pgs degraded (PG_DEGRADED) 2024-03-22T05:46:55.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:55 smithi005 ceph-mon[89904]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 12 pgs inactive, 56 pgs peering) 2024-03-22T05:46:55.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:55 smithi186 ceph-mon[67903]: pgmap v72: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 78/723 objects degraded (10.788%) 2024-03-22T05:46:55.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:55 smithi186 ceph-mon[67903]: Health check failed: Degraded data redundancy: 78/723 objects degraded (10.788%), 22 pgs degraded (PG_DEGRADED) 2024-03-22T05:46:55.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:55 smithi186 ceph-mon[67903]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 12 pgs inactive, 56 pgs peering) 2024-03-22T05:46:56.452 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 22 05:46:56 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-0[97978]: 2024-03-22T05:46:56.209+0000 7f832c70d7c0 -1 osd.0 0 read_superblock omap replica is missing. 2024-03-22T05:46:56.452 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 22 05:46:56 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-0[97978]: 2024-03-22T05:46:56.244+0000 7f832c70d7c0 -1 osd.0 105 log_to_monitors true 2024-03-22T05:46:56.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:56 smithi186 ceph-mon[67903]: from='osd.0 [v2:172.21.15.5:6802/3555255870,v1:172.21.15.5:6803/3555255870]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-22T05:46:56.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:56 smithi186 ceph-mon[67903]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-22T05:46:56.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:56 smithi005 ceph-mon[84705]: from='osd.0 [v2:172.21.15.5:6802/3555255870,v1:172.21.15.5:6803/3555255870]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-22T05:46:56.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:56 smithi005 ceph-mon[84705]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-22T05:46:56.953 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 22 05:46:56 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-0[97978]: 2024-03-22T05:46:56.550+0000 7f831db65700 -1 osd.0 105 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-22T05:46:56.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:56 smithi005 ceph-mon[89904]: from='osd.0 [v2:172.21.15.5:6802/3555255870,v1:172.21.15.5:6803/3555255870]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-22T05:46:56.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:56 smithi005 ceph-mon[89904]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-22T05:46:57.223 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:46:56 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:46:56] "GET /metrics HTTP/1.1" 200 38137 "" "Prometheus/2.43.0" 2024-03-22T05:46:57.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:57 smithi186 ceph-mon[67903]: pgmap v73: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 78/723 objects degraded (10.788%) 2024-03-22T05:46:57.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:57 smithi186 ceph-mon[67903]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-22T05:46:57.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:57 smithi186 ceph-mon[67903]: osdmap e108: 8 total, 7 up, 8 in 2024-03-22T05:46:57.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:57 smithi186 ceph-mon[67903]: from='osd.0 [v2:172.21.15.5:6802/3555255870,v1:172.21.15.5:6803/3555255870]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-03-22T05:46:57.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:57 smithi186 ceph-mon[67903]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-03-22T05:46:57.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:57 smithi005 ceph-mon[84705]: pgmap v73: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 78/723 objects degraded (10.788%) 2024-03-22T05:46:57.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:57 smithi005 ceph-mon[84705]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-22T05:46:57.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:57 smithi005 ceph-mon[84705]: osdmap e108: 8 total, 7 up, 8 in 2024-03-22T05:46:57.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:57 smithi005 ceph-mon[84705]: from='osd.0 [v2:172.21.15.5:6802/3555255870,v1:172.21.15.5:6803/3555255870]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-03-22T05:46:57.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:57 smithi005 ceph-mon[84705]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-03-22T05:46:57.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:57 smithi005 ceph-mon[89904]: pgmap v73: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 78/723 objects degraded (10.788%) 2024-03-22T05:46:57.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:57 smithi005 ceph-mon[89904]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-22T05:46:57.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:57 smithi005 ceph-mon[89904]: osdmap e108: 8 total, 7 up, 8 in 2024-03-22T05:46:57.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:57 smithi005 ceph-mon[89904]: from='osd.0 [v2:172.21.15.5:6802/3555255870,v1:172.21.15.5:6803/3555255870]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-03-22T05:46:57.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:57 smithi005 ceph-mon[89904]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-03-22T05:46:58.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:58 smithi186 ceph-mon[67903]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-22T05:46:58.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:58 smithi186 ceph-mon[67903]: osd.0 [v2:172.21.15.5:6802/3555255870,v1:172.21.15.5:6803/3555255870] boot 2024-03-22T05:46:58.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:58 smithi186 ceph-mon[67903]: osdmap e109: 8 total, 8 up, 8 in 2024-03-22T05:46:58.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:58 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T05:46:58.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:58 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:46:58.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:58 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:46:58.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:58 smithi186 ceph-mon[67903]: osdmap e110: 8 total, 8 up, 8 in 2024-03-22T05:46:58.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:58 smithi005 ceph-mon[89904]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-22T05:46:58.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:58 smithi005 ceph-mon[89904]: osd.0 [v2:172.21.15.5:6802/3555255870,v1:172.21.15.5:6803/3555255870] boot 2024-03-22T05:46:58.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:58 smithi005 ceph-mon[89904]: osdmap e109: 8 total, 8 up, 8 in 2024-03-22T05:46:58.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:58 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T05:46:58.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:58 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:46:58.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:58 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:46:58.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:58 smithi005 ceph-mon[89904]: osdmap e110: 8 total, 8 up, 8 in 2024-03-22T05:46:58.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:58 smithi005 ceph-mon[84705]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-22T05:46:58.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:58 smithi005 ceph-mon[84705]: osd.0 [v2:172.21.15.5:6802/3555255870,v1:172.21.15.5:6803/3555255870] boot 2024-03-22T05:46:58.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:58 smithi005 ceph-mon[84705]: osdmap e109: 8 total, 8 up, 8 in 2024-03-22T05:46:58.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:58 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-22T05:46:58.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:58 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:46:58.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:58 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:46:58.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:58 smithi005 ceph-mon[84705]: osdmap e110: 8 total, 8 up, 8 in 2024-03-22T05:46:59.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:59 smithi005 ceph-mon[84705]: pgmap v76: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 78/723 objects degraded (10.788%) 2024-03-22T05:46:59.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:59 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:46:59.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:46:59 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:46:59.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:59 smithi005 ceph-mon[89904]: pgmap v76: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 78/723 objects degraded (10.788%) 2024-03-22T05:46:59.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:59 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:46:59.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:46:59 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:47:00.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:59 smithi186 ceph-mon[67903]: pgmap v76: 161 pgs: 22 active+undersized+degraded, 36 active+undersized, 103 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 78/723 objects degraded (10.788%) 2024-03-22T05:47:00.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:59 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:47:00.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:46:59 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:47:00.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:00 smithi005 ceph-mon[84705]: Health check update: Degraded data redundancy: 34/723 objects degraded (4.703%), 9 pgs degraded (PG_DEGRADED) 2024-03-22T05:47:00.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:00 smithi005 ceph-mon[89904]: Health check update: Degraded data redundancy: 34/723 objects degraded (4.703%), 9 pgs degraded (PG_DEGRADED) 2024-03-22T05:47:01.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:00 smithi186 ceph-mon[67903]: Health check update: Degraded data redundancy: 34/723 objects degraded (4.703%), 9 pgs degraded (PG_DEGRADED) 2024-03-22T05:47:02.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:01 smithi186 ceph-mon[67903]: pgmap v78: 161 pgs: 37 peering, 9 active+undersized+degraded, 12 active+undersized, 103 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 34/723 objects degraded (4.703%) 2024-03-22T05:47:02.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:01 smithi005 ceph-mon[89904]: pgmap v78: 161 pgs: 37 peering, 9 active+undersized+degraded, 12 active+undersized, 103 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 34/723 objects degraded (4.703%) 2024-03-22T05:47:02.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:01 smithi005 ceph-mon[84705]: pgmap v78: 161 pgs: 37 peering, 9 active+undersized+degraded, 12 active+undersized, 103 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 34/723 objects degraded (4.703%) 2024-03-22T05:47:03.415 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:03 smithi005 ceph-mon[84705]: pgmap v79: 161 pgs: 37 peering, 124 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:47:03.416 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:03 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:47:03.416 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:03 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:47:03.416 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:03 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:47:03.416 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:03 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:47:03.416 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:03 smithi005 ceph-mon[84705]: pgmap v80: 161 pgs: 37 peering, 124 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 872 B/s rd, 0 op/s 2024-03-22T05:47:03.416 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:03 smithi005 ceph-mon[84705]: pgmap v81: 161 pgs: 37 peering, 124 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-22T05:47:03.416 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:03 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:47:03.416 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:03 smithi005 ceph-mon[84705]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:47:03.416 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:03 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:47:03.416 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:03 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:47:03.416 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:03 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:47:03.416 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:03 smithi005 ceph-mon[84705]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:47:03.416 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:03 smithi005 ceph-mon[84705]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 34/723 objects degraded (4.703%), 9 pgs degraded) 2024-03-22T05:47:03.417 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:03 smithi005 ceph-mon[89904]: pgmap v79: 161 pgs: 37 peering, 124 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:47:03.417 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:03 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:47:03.417 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:03 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:47:03.417 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:03 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:47:03.417 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:03 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:47:03.417 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:03 smithi005 ceph-mon[89904]: pgmap v80: 161 pgs: 37 peering, 124 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 872 B/s rd, 0 op/s 2024-03-22T05:47:03.417 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:03 smithi005 ceph-mon[89904]: pgmap v81: 161 pgs: 37 peering, 124 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-22T05:47:03.417 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:03 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:47:03.417 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:03 smithi005 ceph-mon[89904]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:47:03.417 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:03 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:47:03.417 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:03 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:47:03.417 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:03 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:47:03.418 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:03 smithi005 ceph-mon[89904]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:47:03.418 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:03 smithi005 ceph-mon[89904]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 34/723 objects degraded (4.703%), 9 pgs degraded) 2024-03-22T05:47:03.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:03 smithi186 ceph-mon[67903]: pgmap v79: 161 pgs: 37 peering, 124 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:47:03.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:03 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:47:03.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:03 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:47:03.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:03 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:47:03.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:03 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:47:03.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:03 smithi186 ceph-mon[67903]: pgmap v80: 161 pgs: 37 peering, 124 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 872 B/s rd, 0 op/s 2024-03-22T05:47:03.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:03 smithi186 ceph-mon[67903]: pgmap v81: 161 pgs: 37 peering, 124 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-22T05:47:03.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:03 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:47:03.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:03 smithi186 ceph-mon[67903]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:47:03.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:03 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:47:03.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:03 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:47:03.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:03 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:47:03.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:03 smithi186 ceph-mon[67903]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:47:03.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:03 smithi186 ceph-mon[67903]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 34/723 objects degraded (4.703%), 9 pgs degraded) 2024-03-22T05:47:04.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:04 smithi005 ceph-mon[89904]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:47:04.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:04 smithi005 ceph-mon[89904]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:47:04.452 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:47:04.453 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:47:04.453 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:47:04.453 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:47:04.453 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:47:04.453 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:47:04.453 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:47:04.453 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:47:04.453 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:47:04.453 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:47:04.453 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:47:04.453 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:47:04.453 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:47:04.453 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:47:04.453 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:47:04.453 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:47:04.454 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:47:04.454 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:47:04.454 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:47:04.454 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:47:04.454 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:47:04.454 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:47:04.454 INFO:journalctl@ceph.mon.c.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:47:04.454 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:47:04.454 INFO:journalctl@ceph.mon.c.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:47:04.454 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:47:04.454 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:47:04.454 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:47:04.454 INFO:journalctl@ceph.mon.c.smithi005.stdout: return future.result(timeout) 2024-03-22T05:47:04.455 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:47:04.455 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.__get_result() 2024-03-22T05:47:04.455 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:47:04.455 INFO:journalctl@ceph.mon.c.smithi005.stdout: raise self._exception 2024-03-22T05:47:04.455 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:47:04.455 INFO:journalctl@ceph.mon.c.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:47:04.455 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:47:04.455 INFO:journalctl@ceph.mon.c.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:47:04.455 INFO:journalctl@ceph.mon.c.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:47:04.455 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:47:04.455 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:47:04.455 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:47:04.455 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:47:04.455 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:47:04.455 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:47:04.455 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:47:04.456 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:47:04.456 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:47:04.456 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:47:04.456 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:47:04.456 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:47:04.456 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:47:04.456 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:47:04.456 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:47:04.456 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:04 smithi005 ceph-mon[84705]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:47:04.456 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:04 smithi005 ceph-mon[84705]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:47:04.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:47:04.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:47:04.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:47:04.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:47:04.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:47:04.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:47:04.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:47:04.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:47:04.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:47:04.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:47:04.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:47:04.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:47:04.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:47:04.458 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:47:04.458 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:47:04.458 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:47:04.458 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:47:04.458 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:47:04.458 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:47:04.458 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:47:04.458 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:47:04.458 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:47:04.458 INFO:journalctl@ceph.mon.a.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:47:04.458 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:47:04.458 INFO:journalctl@ceph.mon.a.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:47:04.459 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:47:04.459 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:47:04.459 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:47:04.459 INFO:journalctl@ceph.mon.a.smithi005.stdout: return future.result(timeout) 2024-03-22T05:47:04.459 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:47:04.459 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.__get_result() 2024-03-22T05:47:04.459 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:47:04.459 INFO:journalctl@ceph.mon.a.smithi005.stdout: raise self._exception 2024-03-22T05:47:04.459 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:47:04.459 INFO:journalctl@ceph.mon.a.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:47:04.459 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:47:04.459 INFO:journalctl@ceph.mon.a.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:47:04.459 INFO:journalctl@ceph.mon.a.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:47:04.459 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:47:04.460 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:47:04.460 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:47:04.460 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:47:04.460 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:47:04.460 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:47:04.460 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:47:04.460 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:47:04.460 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:47:04.460 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:47:04.460 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:47:04.460 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:47:04.460 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:47:04.460 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:47:04.460 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:47:04.461 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:47:04.461 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:47:04.461 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:47:04.461 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:47:04.461 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:47:04.461 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:04 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:47:04.461 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:04 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:47:04.461 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:04 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:47:04.461 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:04 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:47:04.461 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:04 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-22T05:47:04.462 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:04 smithi005 ceph-mon[89904]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-22T05:47:04.462 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:04 smithi005 ceph-mon[89904]: Upgrade: unsafe to stop osd(s) at this time (14 PGs are or would become offline) 2024-03-22T05:47:04.462 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:47:04.462 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:47:04.462 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:47:04.462 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:47:04.462 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:47:04.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:04 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:47:04.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:04 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:47:04.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:04 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:47:04.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:04 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:47:04.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:04 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-22T05:47:04.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:04 smithi005 ceph-mon[84705]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-22T05:47:04.463 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:04 smithi005 ceph-mon[84705]: Upgrade: unsafe to stop osd(s) at this time (14 PGs are or would become offline) 2024-03-22T05:47:04.463 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:47:04.290+0000 7f7f0ecc9700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:47:04.463 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:47:04.463 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:47:04.464 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:47:04.464 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:47:04.464 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:47:04.464 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:47:04.464 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:47:04.464 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:47:04.464 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:47:04.464 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:47:04.464 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:47:04.464 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:47:04.464 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:47:04.464 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:47:04.464 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:47:04.464 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:47:04.464 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:47:04.465 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:47:04.465 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:47:04.465 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:47:04.465 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:47:04.465 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:47:04.465 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:47:04.465 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:47:04.465 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:47:04.465 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:47:04.465 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:47:04.466 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:47:04.466 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return future.result(timeout) 2024-03-22T05:47:04.466 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:47:04.466 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.__get_result() 2024-03-22T05:47:04.466 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:47:04.466 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: raise self._exception 2024-03-22T05:47:04.466 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:47:04.466 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: config_blobs=daemon_spec.final_config, 2024-03-22T05:47:04.466 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:47:04.467 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:47:04.467 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:47:04.467 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:47:04.467 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:47:04.467 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:47:04.467 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:47:04.467 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:47:04.467 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:47:04.467 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:47:04.467 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:47:04.467 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:47:04.467 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:47:04.467 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:47:04.467 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:47:04.467 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:47:04.467 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:47:04.468 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:47:04.468 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:47:04.468 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:47:04.468 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:47:04.468 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:47:04.468 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:47:04.468 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:04 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:47:04.304+0000 7f7f3e571700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (14 PGs are or would become offline) 2024-03-22T05:47:04.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:04 smithi186 ceph-mon[67903]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:47:04.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:04 smithi186 ceph-mon[67903]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:47:04.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:47:04.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:47:04.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:47:04.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:47:04.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:47:04.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:47:04.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:47:04.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:47:04.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:47:04.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:47:04.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:47:04.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:47:04.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:47:04.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:47:04.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:47:04.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:47:04.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:47:04.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:47:04.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:47:04.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:47:04.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:47:04.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:47:04.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:47:04.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:47:04.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:47:04.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:47:04.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:47:04.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:47:04.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: return future.result(timeout) 2024-03-22T05:47:04.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:47:04.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.__get_result() 2024-03-22T05:47:04.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:47:04.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: raise self._exception 2024-03-22T05:47:04.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:47:04.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:47:04.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:47:04.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:47:04.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:47:04.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:47:04.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:47:04.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:47:04.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:47:04.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:47:04.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:47:04.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:47:04.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:47:04.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:47:04.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:47:04.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:47:04.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:47:04.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:47:04.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:47:04.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:47:04.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:47:04.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:47:04.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:47:04.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:47:04.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:47:04.798 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:04 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:47:04.798 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:04 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:47:04.798 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:04 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:47:04.799 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:04 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:47:04.799 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:04 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-22T05:47:04.799 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:04 smithi186 ceph-mon[67903]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-22T05:47:04.799 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:04 smithi186 ceph-mon[67903]: Upgrade: unsafe to stop osd(s) at this time (14 PGs are or would become offline) 2024-03-22T05:47:05.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:05 smithi005 ceph-mon[89904]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-22T05:47:05.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:05 smithi005 ceph-mon[84705]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-22T05:47:05.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:05 smithi186 ceph-mon[67903]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-22T05:47:07.438 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:47:06] "GET /metrics HTTP/1.1" 200 38137 "" "Prometheus/2.43.0" 2024-03-22T05:47:07.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:07 smithi005 ceph-mon[84705]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 651 B/s rd, 0 op/s 2024-03-22T05:47:07.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:07 smithi005 ceph-mon[89904]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 651 B/s rd, 0 op/s 2024-03-22T05:47:07.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:07 smithi186 ceph-mon[67903]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 651 B/s rd, 0 op/s 2024-03-22T05:47:08.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:08 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:47:08.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:08 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:47:08.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:08 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:47:08.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:08 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:47:08.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:08 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:47:08.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:08 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:47:09.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:09 smithi186 ceph-mon[67903]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 638 B/s rd, 0 op/s 2024-03-22T05:47:09.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:09 smithi005 ceph-mon[84705]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 638 B/s rd, 0 op/s 2024-03-22T05:47:09.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:09 smithi005 ceph-mon[89904]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 638 B/s rd, 0 op/s 2024-03-22T05:47:11.346 INFO:teuthology.orchestra.run.smithi005.stdout:true 2024-03-22T05:47:11.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:11 smithi005 ceph-mon[84705]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:47:11.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:11 smithi005 ceph-mon[89904]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:47:11.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:11 smithi186 ceph-mon[67903]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:47:12.018 INFO:teuthology.orchestra.run.smithi005.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T05:47:12.019 INFO:teuthology.orchestra.run.smithi005.stdout:alertmanager.a smithi005 *:9093,9094 running (4m) 14s ago 26m 28.0M - 0.25.0 c8568f914cd2 ee99dfc435e8 2024-03-22T05:47:12.019 INFO:teuthology.orchestra.run.smithi005.stdout:grafana.a smithi186 *:3000 running (4m) 2m ago 26m 81.5M - 9.4.12 2bacad6d85d8 16fd57dbbd71 2024-03-22T05:47:12.019 INFO:teuthology.orchestra.run.smithi005.stdout:iscsi.foo.smithi005.mcdwbp smithi005 unknown 14s ago 25m 64.5M - 3.5 e1d6a67b021e 5215e03bf81b 2024-03-22T05:47:12.019 INFO:teuthology.orchestra.run.smithi005.stdout:mgr.x smithi186 *:8443,9283,8765 running (5m) 2m ago 31m 448M - 19.0.0-2299-gea01217f aa461e7c5e7f c556421a73f3 2024-03-22T05:47:12.019 INFO:teuthology.orchestra.run.smithi005.stdout:mgr.y smithi005 *:8443,9283,8765 running (15m) 14s ago 32m 505M - 19.0.0-2299-gea01217f aa461e7c5e7f 62a813eebc0d 2024-03-22T05:47:12.019 INFO:teuthology.orchestra.run.smithi005.stdout:mon.a smithi005 running (2m) 14s ago 32m 47.7M 2048M 19.0.0-2299-gea01217f aa461e7c5e7f 111a2aa24dd9 2024-03-22T05:47:12.019 INFO:teuthology.orchestra.run.smithi005.stdout:mon.b smithi186 running (3m) 2m ago 31m 39.1M 2048M 19.0.0-2299-gea01217f aa461e7c5e7f c1798feac15b 2024-03-22T05:47:12.019 INFO:teuthology.orchestra.run.smithi005.stdout:mon.c smithi005 running (2m) 14s ago 31m 36.8M 2048M 19.0.0-2299-gea01217f aa461e7c5e7f 708c72ff74f5 2024-03-22T05:47:12.019 INFO:teuthology.orchestra.run.smithi005.stdout:node-exporter.a smithi005 *:9100 running (5m) 14s ago 27m 16.8M - 1.5.0 0da6a335fe13 f635cd72b9d3 2024-03-22T05:47:12.020 INFO:teuthology.orchestra.run.smithi005.stdout:node-exporter.b smithi186 *:9100 running (5m) 2m ago 27m 15.1M - 1.5.0 0da6a335fe13 d7e0b3d20191 2024-03-22T05:47:12.020 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0 smithi005 running (19s) 14s ago 30m 11.7M 2993M 19.0.0-2299-gea01217f aa461e7c5e7f 31b6cdee5f5b 2024-03-22T05:47:12.020 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1 smithi005 running (30m) 14s ago 30m 57.2M 2993M 17.2.0 e1d6a67b021e a1e7ee1017b6 2024-03-22T05:47:12.020 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2 smithi005 running (30m) 14s ago 30m 49.5M 2993M 17.2.0 e1d6a67b021e 462ea63ad0e5 2024-03-22T05:47:12.020 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3 smithi005 running (29m) 14s ago 29m 56.8M 2993M 17.2.0 e1d6a67b021e b25432a7239b 2024-03-22T05:47:12.020 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4 smithi186 running (29m) 2m ago 29m 56.4M 3248M 17.2.0 e1d6a67b021e cd963671f340 2024-03-22T05:47:12.020 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5 smithi186 running (28m) 2m ago 28m 51.7M 3248M 17.2.0 e1d6a67b021e f98c87f1a2cd 2024-03-22T05:47:12.020 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6 smithi186 running (28m) 2m ago 28m 52.0M 3248M 17.2.0 e1d6a67b021e 704ac49c68c4 2024-03-22T05:47:12.020 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7 smithi186 running (28m) 2m ago 28m 54.3M 3248M 17.2.0 e1d6a67b021e f415b90edf63 2024-03-22T05:47:12.020 INFO:teuthology.orchestra.run.smithi005.stdout:prometheus.a smithi186 *:9095 running (5m) 2m ago 27m 67.2M - 2.43.0 a07b618ecd1d a5a043588d6d 2024-03-22T05:47:12.020 INFO:teuthology.orchestra.run.smithi005.stdout:rgw.foo.smithi005.exdhqu smithi005 *:8000 running (25m) 14s ago 25m 80.9M - 17.2.0 e1d6a67b021e 6bb27fe11553 2024-03-22T05:47:12.020 INFO:teuthology.orchestra.run.smithi005.stdout:rgw.foo.smithi186.mcgodk smithi186 *:8000 running (25m) 2m ago 25m 78.2M - 17.2.0 e1d6a67b021e 19b5c20a34f7 2024-03-22T05:47:12.431 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-03-22T05:47:12.431 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": { 2024-03-22T05:47:12.431 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev)": 3 2024-03-22T05:47:12.431 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:47:12.431 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr": { 2024-03-22T05:47:12.431 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev)": 2 2024-03-22T05:47:12.431 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:47:12.431 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": { 2024-03-22T05:47:12.431 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7, 2024-03-22T05:47:12.432 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev)": 1 2024-03-22T05:47:12.432 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:47:12.432 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw": { 2024-03-22T05:47:12.432 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-22T05:47:12.432 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:47:12.432 INFO:teuthology.orchestra.run.smithi005.stdout: "overall": { 2024-03-22T05:47:12.432 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 9, 2024-03-22T05:47:12.432 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev)": 6 2024-03-22T05:47:12.432 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-03-22T05:47:12.432 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-03-22T05:47:12.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:12 smithi005 ceph-mon[84705]: from='client.54282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:47:12.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:12 smithi005 ceph-mon[84705]: from='client.54288 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:47:12.705 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:12 smithi005 ceph-mon[84705]: from='client.54294 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:47:12.705 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:12 smithi005 ceph-mon[84705]: from='client.? 172.21.15.5:0/2224606300' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:47:12.705 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:12 smithi005 ceph-mon[89904]: from='client.54282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:47:12.705 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:12 smithi005 ceph-mon[89904]: from='client.54288 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:47:12.705 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:12 smithi005 ceph-mon[89904]: from='client.54294 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:47:12.705 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:12 smithi005 ceph-mon[89904]: from='client.? 172.21.15.5:0/2224606300' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:47:12.776 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-03-22T05:47:12.777 INFO:teuthology.orchestra.run.smithi005.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9", 2024-03-22T05:47:12.777 INFO:teuthology.orchestra.run.smithi005.stdout: "in_progress": true, 2024-03-22T05:47:12.777 INFO:teuthology.orchestra.run.smithi005.stdout: "which": "Upgrading daemons of type(s) osd. Upgrade limited to 2 daemons (1 remaining).", 2024-03-22T05:47:12.777 INFO:teuthology.orchestra.run.smithi005.stdout: "services_complete": [], 2024-03-22T05:47:12.777 INFO:teuthology.orchestra.run.smithi005.stdout: "progress": "1/8 daemons upgraded", 2024-03-22T05:47:12.777 INFO:teuthology.orchestra.run.smithi005.stdout: "message": "Currently upgrading osd daemons", 2024-03-22T05:47:12.777 INFO:teuthology.orchestra.run.smithi005.stdout: "is_paused": false 2024-03-22T05:47:12.777 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-03-22T05:47:12.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:12 smithi186 ceph-mon[67903]: from='client.54282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:47:12.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:12 smithi186 ceph-mon[67903]: from='client.54288 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:47:12.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:12 smithi186 ceph-mon[67903]: from='client.54294 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:47:12.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:12 smithi186 ceph-mon[67903]: from='client.? 172.21.15.5:0/2224606300' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:47:13.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:13 smithi186 ceph-mon[67903]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:47:13.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:13 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:47:13.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:13 smithi186 ceph-mon[67903]: from='client.64240 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:47:13.880 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:13 smithi005 ceph-mon[84705]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:47:13.880 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:13 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:47:13.880 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:13 smithi005 ceph-mon[84705]: from='client.64240 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:47:13.881 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:13 smithi005 ceph-mon[89904]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:47:13.881 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:13 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:47:13.881 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:13 smithi005 ceph-mon[89904]: from='client.64240 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:47:14.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:14 smithi005 ceph-mon[84705]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:47:14.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:14 smithi005 ceph-mon[89904]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:47:15.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:14 smithi186 ceph-mon[67903]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:47:15.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:15 smithi005 ceph-mon[84705]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:47:15.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:15 smithi005 ceph-mon[89904]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:47:16.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:15 smithi186 ceph-mon[67903]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:47:17.452 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:16 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:47:16] "GET /metrics HTTP/1.1" 200 38165 "" "Prometheus/2.43.0" 2024-03-22T05:47:17.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:17 smithi005 ceph-mon[89904]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:47:17.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:17 smithi005 ceph-mon[84705]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:47:18.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:17 smithi186 ceph-mon[67903]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:47:19.918 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:19 smithi005 ceph-mon[84705]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:47:19.918 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:19 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-22T05:47:19.918 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:19 smithi005 ceph-mon[84705]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-22T05:47:19.918 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:19 smithi005 ceph-mon[84705]: Upgrade: osd.1 is safe to restart 2024-03-22T05:47:19.919 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:19 smithi005 ceph-mon[89904]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:47:19.919 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:19 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-22T05:47:19.919 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:19 smithi005 ceph-mon[89904]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-22T05:47:19.919 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:19 smithi005 ceph-mon[89904]: Upgrade: osd.1 is safe to restart 2024-03-22T05:47:20.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:19 smithi186 ceph-mon[67903]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:47:20.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:19 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-22T05:47:20.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:19 smithi186 ceph-mon[67903]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-22T05:47:20.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:19 smithi186 ceph-mon[67903]: Upgrade: osd.1 is safe to restart 2024-03-22T05:47:22.157 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:21 smithi005 ceph-mon[84705]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:47:22.157 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:21 smithi005 ceph-mon[84705]: Upgrade: Updating osd.1 2024-03-22T05:47:22.157 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:21 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:47:22.157 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:21 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-22T05:47:22.157 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:21 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:47:22.157 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:21 smithi005 ceph-mon[84705]: Deploying daemon osd.1 on smithi005 2024-03-22T05:47:22.157 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:21 smithi005 ceph-mon[89904]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:47:22.158 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:21 smithi005 ceph-mon[89904]: Upgrade: Updating osd.1 2024-03-22T05:47:22.158 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:21 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:47:22.158 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:21 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-22T05:47:22.158 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:21 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:47:22.158 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:21 smithi005 ceph-mon[89904]: Deploying daemon osd.1 on smithi005 2024-03-22T05:47:22.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:21 smithi186 ceph-mon[67903]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:47:22.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:21 smithi186 ceph-mon[67903]: Upgrade: Updating osd.1 2024-03-22T05:47:22.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:21 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:47:22.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:21 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-22T05:47:22.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:21 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:47:22.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:21 smithi186 ceph-mon[67903]: Deploying daemon osd.1 on smithi005 2024-03-22T05:47:23.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:23 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:47:23.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:23 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:47:23.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:23 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:47:24.511 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 22 05:47:24 smithi005 systemd[1]: Stopping Ceph osd.1 for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:47:24.512 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:24 smithi005 ceph-mon[84705]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:47:24.512 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:24 smithi005 ceph-mon[89904]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:47:24.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:24 smithi186 ceph-mon[67903]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:47:24.952 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 22 05:47:24 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-1[35327]: 2024-03-22T05:47:24.511+0000 7fc780136700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T05:47:24.952 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 22 05:47:24 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-1[35327]: 2024-03-22T05:47:24.511+0000 7fc780136700 -1 osd.1 110 *** Got signal Terminated *** 2024-03-22T05:47:24.952 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 22 05:47:24 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-1[35327]: 2024-03-22T05:47:24.511+0000 7fc780136700 -1 osd.1 110 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-22T05:47:25.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:25 smithi186 ceph-mon[67903]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:47:25.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:25 smithi186 ceph-mon[67903]: osd.1 marked itself down and dead 2024-03-22T05:47:25.666 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:25 smithi005 ceph-mon[84705]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:47:25.666 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:25 smithi005 ceph-mon[84705]: osd.1 marked itself down and dead 2024-03-22T05:47:25.666 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:25 smithi005 ceph-mon[89904]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:47:25.666 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:25 smithi005 ceph-mon[89904]: osd.1 marked itself down and dead 2024-03-22T05:47:25.920 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 22 05:47:25 smithi005 podman[102858]: 2024-03-22 05:47:25.665905364 +0000 UTC m=+1.278166057 container died a1e7ee1017b6430d564829b95eddc54ac3c0dd39cb5eebbcf9bced11f388580f (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-1, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, vcs-type=git, release=754, name=centos-stream, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., io.buildah.version=1.19.8, io.openshift.expose-services=, ceph=True, distribution-scope=public, GIT_BRANCH=HEAD, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, maintainer=Guillaume Abrioux ) 2024-03-22T05:47:26.202 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 22 05:47:26 smithi005 podman[102858]: 2024-03-22 05:47:26.028824944 +0000 UTC m=+1.641085635 container cleanup a1e7ee1017b6430d564829b95eddc54ac3c0dd39cb5eebbcf9bced11f388580f (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-1, architecture=x86_64, release=754, version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , RELEASE=HEAD, GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, com.redhat.component=centos-stream-container, distribution-scope=public, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_CLEAN=True, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8) 2024-03-22T05:47:26.202 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 22 05:47:26 smithi005 bash[102858]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-1 2024-03-22T05:47:26.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:26 smithi186 ceph-mon[67903]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:47:26.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:26 smithi186 ceph-mon[67903]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-22T05:47:26.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:26 smithi186 ceph-mon[67903]: osdmap e111: 8 total, 7 up, 8 in 2024-03-22T05:47:26.597 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:26 smithi005 ceph-mon[84705]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:47:26.597 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:26 smithi005 ceph-mon[84705]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-22T05:47:26.597 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:26 smithi005 ceph-mon[84705]: osdmap e111: 8 total, 7 up, 8 in 2024-03-22T05:47:26.598 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:26 smithi005 ceph-mon[89904]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:47:26.598 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:26 smithi005 ceph-mon[89904]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-22T05:47:26.598 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:26 smithi005 ceph-mon[89904]: osdmap e111: 8 total, 7 up, 8 in 2024-03-22T05:47:26.598 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 22 05:47:26 smithi005 podman[102892]: 2024-03-22 05:47:26.288626183 +0000 UTC m=+0.619420158 container remove a1e7ee1017b6430d564829b95eddc54ac3c0dd39cb5eebbcf9bced11f388580f (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-1, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0, version=8, GIT_CLEAN=True, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, distribution-scope=public, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, name=centos-stream) 2024-03-22T05:47:26.907 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 22 05:47:26 smithi005 podman[102983]: 2024-03-22 05:47:26.500826565 +0000 UTC m=+0.018244231 image pull aa461e7c5e7f322b169c789b389d594b994ea7ffcbc9e24e563a3ec9b69bc3c1 quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9 2024-03-22T05:47:26.908 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 22 05:47:26 smithi005 podman[102983]: 2024-03-22 05:47:26.606039181 +0000 UTC m=+0.123456899 container create d9ef0c087fd6c7977b9ba9fff490981e6d1943c53d4f9f087e41c1ac6ddaefae (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-1-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, ceph=True) 2024-03-22T05:47:26.908 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 22 05:47:26 smithi005 podman[102983]: 2024-03-22 05:47:26.842241579 +0000 UTC m=+0.359659244 container init d9ef0c087fd6c7977b9ba9fff490981e6d1943c53d4f9f087e41c1ac6ddaefae (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-1-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-03-22T05:47:26.908 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 22 05:47:26 smithi005 podman[102983]: 2024-03-22 05:47:26.845565087 +0000 UTC m=+0.362982748 container start d9ef0c087fd6c7977b9ba9fff490981e6d1943c53d4f9f087e41c1ac6ddaefae (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-1-deactivate, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217) 2024-03-22T05:47:26.908 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 22 05:47:26 smithi005 podman[102983]: 2024-03-22 05:47:26.908524153 +0000 UTC m=+0.425941821 container attach d9ef0c087fd6c7977b9ba9fff490981e6d1943c53d4f9f087e41c1ac6ddaefae (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319) 2024-03-22T05:47:27.202 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:26 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:47:26] "GET /metrics HTTP/1.1" 200 38158 "" "Prometheus/2.43.0" 2024-03-22T05:47:27.203 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 22 05:47:27 smithi005 podman[102983]: 2024-03-22 05:47:27.151501261 +0000 UTC m=+0.668918913 container died d9ef0c087fd6c7977b9ba9fff490981e6d1943c53d4f9f087e41c1ac6ddaefae (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-1-deactivate, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True) 2024-03-22T05:47:27.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:27 smithi186 ceph-mon[67903]: osdmap e112: 8 total, 7 up, 8 in 2024-03-22T05:47:27.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:27 smithi005 ceph-mon[84705]: osdmap e112: 8 total, 7 up, 8 in 2024-03-22T05:47:27.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:27 smithi005 ceph-mon[89904]: osdmap e112: 8 total, 7 up, 8 in 2024-03-22T05:47:28.180 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 22 05:47:27 smithi005 podman[103008]: 2024-03-22 05:47:27.754451723 +0000 UTC m=+0.597497763 container remove d9ef0c087fd6c7977b9ba9fff490981e6d1943c53d4f9f087e41c1ac6ddaefae (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-1-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, org.label-schema.build-date=20240319, GIT_CLEAN=True) 2024-03-22T05:47:28.181 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 22 05:47:27 smithi005 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@osd.1.service: Deactivated successfully. 2024-03-22T05:47:28.181 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 22 05:47:27 smithi005 systemd[1]: Stopped Ceph osd.1 for f59f36e2-e80a-11ee-95cd-87774f69a715. 2024-03-22T05:47:28.181 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 22 05:47:27 smithi005 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@osd.1.service: Consumed 15.315s CPU time. 2024-03-22T05:47:28.181 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 22 05:47:27 smithi005 systemd[1]: Starting Ceph osd.1 for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:47:28.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:28 smithi005 ceph-mon[84705]: pgmap v95: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:47:28.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:28 smithi005 ceph-mon[89904]: pgmap v95: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:47:28.453 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 22 05:47:28 smithi005 podman[103113]: 2024-03-22 05:47:28.18048023 +0000 UTC m=+0.106743646 container create 79397acd83e7a453cdf8ed3e609beca64bd37100157dd79ba2ed7263911ecbe1 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-1-activate, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, GIT_BRANCH=HEAD) 2024-03-22T05:47:28.453 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 22 05:47:28 smithi005 podman[103113]: 2024-03-22 05:47:28.09115035 +0000 UTC m=+0.017413750 image pull aa461e7c5e7f322b169c789b389d594b994ea7ffcbc9e24e563a3ec9b69bc3c1 quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9 2024-03-22T05:47:28.453 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 22 05:47:28 smithi005 podman[103113]: 2024-03-22 05:47:28.407944333 +0000 UTC m=+0.334207701 container init 79397acd83e7a453cdf8ed3e609beca64bd37100157dd79ba2ed7263911ecbe1 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-1-activate, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-22T05:47:28.453 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 22 05:47:28 smithi005 podman[103113]: 2024-03-22 05:47:28.411109993 +0000 UTC m=+0.337373369 container start 79397acd83e7a453cdf8ed3e609beca64bd37100157dd79ba2ed7263911ecbe1 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-1-activate, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-22T05:47:28.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:28 smithi186 ceph-mon[67903]: pgmap v95: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:47:28.856 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 22 05:47:28 smithi005 podman[103113]: 2024-03-22 05:47:28.508041554 +0000 UTC m=+0.434304926 container attach 79397acd83e7a453cdf8ed3e609beca64bd37100157dd79ba2ed7263911ecbe1 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-1-activate, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-22T05:47:29.196 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 22 05:47:28 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-1-activate[103129]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-22T05:47:29.196 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 22 05:47:28 smithi005 bash[103113]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-22T05:47:29.196 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 22 05:47:28 smithi005 bash[103113]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-03-22T05:47:29.196 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 22 05:47:28 smithi005 bash[103113]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-03-22T05:47:29.196 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 22 05:47:28 smithi005 bash[103113]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-03-22T05:47:29.196 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 22 05:47:28 smithi005 bash[103113]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-03-22T05:47:29.196 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 22 05:47:28 smithi005 bash[103113]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-22T05:47:29.196 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 22 05:47:28 smithi005 bash[103113]: --> ceph-volume raw activate successful for osd ID: 1 2024-03-22T05:47:29.196 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 22 05:47:28 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-1-activate[103129]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-03-22T05:47:29.196 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 22 05:47:28 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-1-activate[103129]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-03-22T05:47:29.197 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 22 05:47:28 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-1-activate[103129]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-03-22T05:47:29.197 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 22 05:47:28 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-1-activate[103129]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-03-22T05:47:29.197 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 22 05:47:28 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-1-activate[103129]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-22T05:47:29.197 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 22 05:47:28 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-1-activate[103129]: --> ceph-volume raw activate successful for osd ID: 1 2024-03-22T05:47:29.197 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 22 05:47:28 smithi005 podman[103113]: 2024-03-22 05:47:28.881015129 +0000 UTC m=+0.807278505 container died 79397acd83e7a453cdf8ed3e609beca64bd37100157dd79ba2ed7263911ecbe1 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-1-activate, org.label-schema.vendor=CentOS, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-22T05:47:29.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:29 smithi005 ceph-mon[84705]: Health check failed: Degraded data redundancy: 7/723 objects degraded (0.968%), 2 pgs degraded (PG_DEGRADED) 2024-03-22T05:47:29.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:29 smithi005 ceph-mon[89904]: Health check failed: Degraded data redundancy: 7/723 objects degraded (0.968%), 2 pgs degraded (PG_DEGRADED) 2024-03-22T05:47:29.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:29 smithi186 ceph-mon[67903]: Health check failed: Degraded data redundancy: 7/723 objects degraded (0.968%), 2 pgs degraded (PG_DEGRADED) 2024-03-22T05:47:29.895 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 22 05:47:29 smithi005 podman[103263]: 2024-03-22 05:47:29.539132264 +0000 UTC m=+0.653116794 container remove 79397acd83e7a453cdf8ed3e609beca64bd37100157dd79ba2ed7263911ecbe1 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-1-activate, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, io.buildah.version=1.33.5) 2024-03-22T05:47:29.895 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 22 05:47:29 smithi005 podman[103329]: 2024-03-22 05:47:29.659071573 +0000 UTC m=+0.017540031 image pull aa461e7c5e7f322b169c789b389d594b994ea7ffcbc9e24e563a3ec9b69bc3c1 quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9 2024-03-22T05:47:29.895 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 22 05:47:29 smithi005 podman[103329]: 2024-03-22 05:47:29.764848853 +0000 UTC m=+0.123317301 container create 7d21bc9fd3c2343f595dd18fa339b995382d09d3a2464b6173f250dde69c98fa (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-1, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD) 2024-03-22T05:47:30.294 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 22 05:47:29 smithi005 podman[103329]: 2024-03-22 05:47:29.992355558 +0000 UTC m=+0.350824001 container init 7d21bc9fd3c2343f595dd18fa339b995382d09d3a2464b6173f250dde69c98fa (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-1, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-22T05:47:30.295 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 22 05:47:29 smithi005 podman[103329]: 2024-03-22 05:47:29.995422818 +0000 UTC m=+0.353891262 container start 7d21bc9fd3c2343f595dd18fa339b995382d09d3a2464b6173f250dde69c98fa (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-1, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-22T05:47:30.295 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 22 05:47:30 smithi005 ceph-osd[103348]: -- 172.21.15.5:0/3826010953 <== mon.1 v2:172.21.15.5:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x564e2926b680 con 0x564e29234c00 2024-03-22T05:47:30.295 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 22 05:47:30 smithi005 bash[103329]: 7d21bc9fd3c2343f595dd18fa339b995382d09d3a2464b6173f250dde69c98fa 2024-03-22T05:47:30.295 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 22 05:47:30 smithi005 systemd[1]: Started Ceph osd.1 for f59f36e2-e80a-11ee-95cd-87774f69a715. 2024-03-22T05:47:30.295 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:30 smithi005 ceph-mon[84705]: pgmap v96: 161 pgs: 2 active+undersized+degraded, 20 stale+active+clean, 5 active+undersized, 134 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 7/723 objects degraded (0.968%) 2024-03-22T05:47:30.295 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:30 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:47:30.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:30 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:47:30.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:30 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:47:30.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:30 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:47:30.703 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 22 05:47:30 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-1[103344]: 2024-03-22T05:47:30.538+0000 7f67b158a7c0 -1 Falling back to public interface 2024-03-22T05:47:30.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:30 smithi005 ceph-mon[89904]: pgmap v96: 161 pgs: 2 active+undersized+degraded, 20 stale+active+clean, 5 active+undersized, 134 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 7/723 objects degraded (0.968%) 2024-03-22T05:47:30.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:30 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:47:30.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:30 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:47:30.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:30 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:47:30.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:30 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:47:30.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:30 smithi186 ceph-mon[67903]: pgmap v96: 161 pgs: 2 active+undersized+degraded, 20 stale+active+clean, 5 active+undersized, 134 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 7/723 objects degraded (0.968%) 2024-03-22T05:47:30.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:30 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:47:30.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:30 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:47:30.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:30 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:47:30.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:30 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:47:32.616 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:32 smithi005 ceph-mon[84705]: pgmap v97: 161 pgs: 23 active+undersized+degraded, 40 active+undersized, 98 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 103/723 objects degraded (14.246%) 2024-03-22T05:47:32.617 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:32 smithi005 ceph-mon[89904]: pgmap v97: 161 pgs: 23 active+undersized+degraded, 40 active+undersized, 98 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 103/723 objects degraded (14.246%) 2024-03-22T05:47:32.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:32 smithi186 ceph-mon[67903]: pgmap v97: 161 pgs: 23 active+undersized+degraded, 40 active+undersized, 98 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 103/723 objects degraded (14.246%) 2024-03-22T05:47:33.581 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 22 05:47:33 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-1[103344]: 2024-03-22T05:47:33.266+0000 7f67b158a7c0 -1 osd.1 0 read_superblock omap replica is missing. 2024-03-22T05:47:33.581 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 22 05:47:33 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-1[103344]: 2024-03-22T05:47:33.303+0000 7f67b158a7c0 -1 osd.1 110 log_to_monitors true 2024-03-22T05:47:34.702 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 22 05:47:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-1[103344]: 2024-03-22T05:47:34.327+0000 7f67a29e2700 -1 osd.1 110 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-22T05:47:34.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:34 smithi005 ceph-mon[84705]: pgmap v98: 161 pgs: 23 active+undersized+degraded, 40 active+undersized, 98 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 103/723 objects degraded (14.246%) 2024-03-22T05:47:34.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:34 smithi005 ceph-mon[84705]: from='osd.1 [v2:172.21.15.5:6810/659788395,v1:172.21.15.5:6811/659788395]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-22T05:47:34.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:34 smithi005 ceph-mon[84705]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-22T05:47:34.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:34 smithi005 ceph-mon[89904]: pgmap v98: 161 pgs: 23 active+undersized+degraded, 40 active+undersized, 98 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 103/723 objects degraded (14.246%) 2024-03-22T05:47:34.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:34 smithi005 ceph-mon[89904]: from='osd.1 [v2:172.21.15.5:6810/659788395,v1:172.21.15.5:6811/659788395]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-22T05:47:34.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:34 smithi005 ceph-mon[89904]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-22T05:47:34.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:34 smithi186 ceph-mon[67903]: pgmap v98: 161 pgs: 23 active+undersized+degraded, 40 active+undersized, 98 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 103/723 objects degraded (14.246%) 2024-03-22T05:47:34.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:34 smithi186 ceph-mon[67903]: from='osd.1 [v2:172.21.15.5:6810/659788395,v1:172.21.15.5:6811/659788395]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-22T05:47:34.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:34 smithi186 ceph-mon[67903]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-22T05:47:35.308 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:35 smithi005 ceph-mon[84705]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:47:35.309 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:35 smithi005 ceph-mon[84705]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-22T05:47:35.309 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:35 smithi005 ceph-mon[84705]: osdmap e113: 8 total, 7 up, 8 in 2024-03-22T05:47:35.312 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:35 smithi005 ceph-mon[89904]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:47:35.312 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:35 smithi005 ceph-mon[89904]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-22T05:47:35.312 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:35 smithi005 ceph-mon[89904]: osdmap e113: 8 total, 7 up, 8 in 2024-03-22T05:47:35.312 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:35 smithi005 ceph-mon[89904]: from='osd.1 [v2:172.21.15.5:6810/659788395,v1:172.21.15.5:6811/659788395]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-03-22T05:47:35.312 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:35 smithi005 ceph-mon[89904]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-03-22T05:47:35.312 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:35 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:47:35.312 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:35 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:47:35.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:35 smithi005 ceph-mon[84705]: from='osd.1 [v2:172.21.15.5:6810/659788395,v1:172.21.15.5:6811/659788395]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-03-22T05:47:35.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:35 smithi005 ceph-mon[84705]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-03-22T05:47:35.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:35 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:47:35.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:35 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:47:35.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:35 smithi186 ceph-mon[67903]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:47:35.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:35 smithi186 ceph-mon[67903]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-22T05:47:35.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:35 smithi186 ceph-mon[67903]: osdmap e113: 8 total, 7 up, 8 in 2024-03-22T05:47:35.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:35 smithi186 ceph-mon[67903]: from='osd.1 [v2:172.21.15.5:6810/659788395,v1:172.21.15.5:6811/659788395]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-03-22T05:47:35.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:35 smithi186 ceph-mon[67903]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-03-22T05:47:35.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:35 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:47:35.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:35 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:47:36.563 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:36 smithi005 ceph-mon[84705]: pgmap v100: 161 pgs: 23 active+undersized+degraded, 40 active+undersized, 98 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 103/723 objects degraded (14.246%) 2024-03-22T05:47:36.563 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:36 smithi005 ceph-mon[84705]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-22T05:47:36.564 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:36 smithi005 ceph-mon[84705]: osd.1 [v2:172.21.15.5:6810/659788395,v1:172.21.15.5:6811/659788395] boot 2024-03-22T05:47:36.564 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:36 smithi005 ceph-mon[84705]: osdmap e114: 8 total, 8 up, 8 in 2024-03-22T05:47:36.564 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:36 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T05:47:36.564 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:36 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:47:36.564 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:36 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:47:36.564 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:36 smithi005 ceph-mon[89904]: pgmap v100: 161 pgs: 23 active+undersized+degraded, 40 active+undersized, 98 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 103/723 objects degraded (14.246%) 2024-03-22T05:47:36.564 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:36 smithi005 ceph-mon[89904]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-22T05:47:36.564 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:36 smithi005 ceph-mon[89904]: osd.1 [v2:172.21.15.5:6810/659788395,v1:172.21.15.5:6811/659788395] boot 2024-03-22T05:47:36.564 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:36 smithi005 ceph-mon[89904]: osdmap e114: 8 total, 8 up, 8 in 2024-03-22T05:47:36.564 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:36 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T05:47:36.564 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:36 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:47:36.564 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:36 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:47:36.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:36 smithi186 ceph-mon[67903]: pgmap v100: 161 pgs: 23 active+undersized+degraded, 40 active+undersized, 98 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 103/723 objects degraded (14.246%) 2024-03-22T05:47:36.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:36 smithi186 ceph-mon[67903]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-22T05:47:36.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:36 smithi186 ceph-mon[67903]: osd.1 [v2:172.21.15.5:6810/659788395,v1:172.21.15.5:6811/659788395] boot 2024-03-22T05:47:36.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:36 smithi186 ceph-mon[67903]: osdmap e114: 8 total, 8 up, 8 in 2024-03-22T05:47:36.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:36 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-22T05:47:36.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:36 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:47:36.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:36 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:47:37.202 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:36 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:47:36] "GET /metrics HTTP/1.1" 200 38158 "" "Prometheus/2.43.0" 2024-03-22T05:47:37.546 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:37 smithi005 ceph-mon[89904]: osdmap e115: 8 total, 8 up, 8 in 2024-03-22T05:47:37.546 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:37 smithi005 ceph-mon[89904]: Health check update: Degraded data redundancy: 103/723 objects degraded (14.246%), 23 pgs degraded (PG_DEGRADED) 2024-03-22T05:47:37.546 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:37 smithi005 ceph-mon[84705]: osdmap e115: 8 total, 8 up, 8 in 2024-03-22T05:47:37.546 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:37 smithi005 ceph-mon[84705]: Health check update: Degraded data redundancy: 103/723 objects degraded (14.246%), 23 pgs degraded (PG_DEGRADED) 2024-03-22T05:47:37.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:37 smithi186 ceph-mon[67903]: osdmap e115: 8 total, 8 up, 8 in 2024-03-22T05:47:37.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:37 smithi186 ceph-mon[67903]: Health check update: Degraded data redundancy: 103/723 objects degraded (14.246%), 23 pgs degraded (PG_DEGRADED) 2024-03-22T05:47:38.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:38 smithi005 ceph-mon[84705]: pgmap v103: 161 pgs: 23 active+undersized+degraded, 40 active+undersized, 98 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 103/723 objects degraded (14.246%) 2024-03-22T05:47:38.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:38 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:47:38.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:38 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:47:38.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:38 smithi005 ceph-mon[89904]: pgmap v103: 161 pgs: 23 active+undersized+degraded, 40 active+undersized, 98 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 103/723 objects degraded (14.246%) 2024-03-22T05:47:38.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:38 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:47:38.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:38 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:47:38.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:38 smithi186 ceph-mon[67903]: pgmap v103: 161 pgs: 23 active+undersized+degraded, 40 active+undersized, 98 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 103/723 objects degraded (14.246%) 2024-03-22T05:47:38.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:38 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:47:38.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:38 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:47:40.179 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-mon[84705]: pgmap v104: 161 pgs: 22 active+undersized+degraded, 35 active+undersized, 104 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 97/723 objects degraded (13.416%) 2024-03-22T05:47:40.180 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:47:40.180 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:47:40.180 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:47:40.180 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:47:40.180 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-mon[84705]: pgmap v105: 161 pgs: 8 active+undersized+degraded, 13 active+undersized, 140 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 40/723 objects degraded (5.533%) 2024-03-22T05:47:40.180 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-mon[84705]: pgmap v106: 161 pgs: 8 active+undersized+degraded, 13 active+undersized, 140 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 40/723 objects degraded (5.533%) 2024-03-22T05:47:40.180 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:47:40.180 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-mon[84705]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:47:40.180 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:47:40.180 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:47:40.180 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:47:40.180 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-mon[84705]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:47:40.181 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-mon[89904]: pgmap v104: 161 pgs: 22 active+undersized+degraded, 35 active+undersized, 104 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 97/723 objects degraded (13.416%) 2024-03-22T05:47:40.181 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:47:40.181 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:47:40.181 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:47:40.181 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:47:40.181 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-mon[89904]: pgmap v105: 161 pgs: 8 active+undersized+degraded, 13 active+undersized, 140 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 40/723 objects degraded (5.533%) 2024-03-22T05:47:40.181 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-mon[89904]: pgmap v106: 161 pgs: 8 active+undersized+degraded, 13 active+undersized, 140 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 40/723 objects degraded (5.533%) 2024-03-22T05:47:40.181 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:47:40.181 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-mon[89904]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:47:40.181 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:47:40.181 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:47:40.181 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:47:40.181 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-mon[89904]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:47:40.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:40 smithi186 ceph-mon[67903]: pgmap v104: 161 pgs: 22 active+undersized+degraded, 35 active+undersized, 104 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 97/723 objects degraded (13.416%) 2024-03-22T05:47:40.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:40 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:47:40.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:40 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:47:40.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:40 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:47:40.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:40 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:47:40.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:40 smithi186 ceph-mon[67903]: pgmap v105: 161 pgs: 8 active+undersized+degraded, 13 active+undersized, 140 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 40/723 objects degraded (5.533%) 2024-03-22T05:47:40.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:40 smithi186 ceph-mon[67903]: pgmap v106: 161 pgs: 8 active+undersized+degraded, 13 active+undersized, 140 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 40/723 objects degraded (5.533%) 2024-03-22T05:47:40.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:40 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:47:40.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:40 smithi186 ceph-mon[67903]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:47:40.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:40 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:47:40.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:40 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:47:40.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:40 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:47:40.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:40 smithi186 ceph-mon[67903]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:47:41.143 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:47:40.963+0000 7f7f0ecc9700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:47:41.143 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:47:41.143 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:47:41.143 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:47:41.143 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:47:41.144 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:47:41.144 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:47:41.144 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:47:41.144 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:47:41.144 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:47:41.144 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:47:41.144 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:47:41.144 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:47:41.144 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:47:41.144 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:47:41.144 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:47:41.144 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:47:41.144 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:47:41.144 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:47:41.145 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:47:41.145 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:47:41.145 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:47:41.145 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:47:41.145 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:47:41.145 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:47:41.145 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:47:41.146 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:47:41.146 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:47:41.146 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:47:41.146 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return future.result(timeout) 2024-03-22T05:47:41.146 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:47:41.146 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.__get_result() 2024-03-22T05:47:41.146 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:47:41.146 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: raise self._exception 2024-03-22T05:47:41.146 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:47:41.146 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: config_blobs=daemon_spec.final_config, 2024-03-22T05:47:41.146 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:47:41.146 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:47:41.146 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:47:41.146 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:47:41.146 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:47:41.146 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:47:41.147 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:47:41.147 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:47:41.147 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:47:41.147 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:47:41.147 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:47:41.147 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:47:41.147 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:47:41.147 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:47:41.147 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:47:41.147 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:47:41.147 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:47:41.147 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:47:41.147 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:47:41.147 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:47:41.147 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:47:41.148 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:47:41.148 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:47:41.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:47:41.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:47:41.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:47:41.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:47:41.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:47:41.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:47:41.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:47:41.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:47:41.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:47:41.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:47:41.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-22T05:47:41.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-22T05:47:41.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-22T05:47:41.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:47:41.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:47:41.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-22T05:47:41.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-22T05:47:41.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-22T05:47:41.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-22T05:47:41.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-22T05:47:41.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-22T05:47:41.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-22T05:47:41.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-22T05:47:41.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-22T05:47:41.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-22T05:47:41.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-22T05:47:41.455 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:47:41.455 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:47:41.455 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:47:41.455 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:47:41.455 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:47:41.455 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:47:41.455 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:47:41.455 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:47:41.455 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:47:41.455 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:47:41.456 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-22T05:47:41.456 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-22T05:47:41.456 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-22T05:47:41.456 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:47:41.456 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:47:41.456 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-22T05:47:41.456 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-22T05:47:41.456 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-22T05:47:41.456 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-22T05:47:41.456 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-22T05:47:41.456 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-22T05:47:41.456 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-22T05:47:41.456 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-22T05:47:41.456 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-22T05:47:41.456 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-22T05:47:41.457 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-22T05:47:41.457 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-22T05:47:41.457 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:47:41.457 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:47:41.457 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-22T05:47:41.457 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-22T05:47:41.457 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-22T05:47:41.457 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-22T05:47:41.457 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-22T05:47:41.457 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-22T05:47:41.457 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-22T05:47:41.457 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-22T05:47:41.457 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-22T05:47:41.457 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:47:41.457 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:47:41.458 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:47:41.458 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:47:41.458 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:47:41.458 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:47:41.458 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:47:41.458 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:47:41.458 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:47:41.458 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:47:41.458 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:47:41.458 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-22T05:47:41.459 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:47:41.459 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:47:41.459 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-22T05:47:41.459 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-22T05:47:41.459 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-22T05:47:41.459 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-22T05:47:41.459 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-22T05:47:41.459 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-22T05:47:41.459 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-22T05:47:41.459 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-22T05:47:41.459 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-22T05:47:41.459 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:47:41.459 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:47:41.459 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:47:41.459 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:47:41.460 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:47:41.460 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:47:41.460 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:47:41.460 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:47:41.460 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:47:41.460 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:47:41.460 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:47:41.460 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:47:41.460 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-22T05:47:41.460 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-22T05:47:41.460 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-22T05:47:41.461 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:47:41.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:47:41.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-22T05:47:41.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-22T05:47:41.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-22T05:47:41.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:41 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:47:41.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:41 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:47:41.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:41 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:47:41.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:41 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:47:41.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:41 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:47:41.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:41 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:47:41.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:41 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:47:41.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:41 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:47:41.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:41 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:47:41.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:41 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:47:41.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:41 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:47:41.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:41 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-22T05:47:41.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:41 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-22T05:47:41.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:41 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-22T05:47:41.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:41 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:47:41.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:41 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:47:41.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:41 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-22T05:47:41.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:41 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-22T05:47:41.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:41 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-22T05:47:41.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:41 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-22T05:47:41.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:41 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-22T05:47:41.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:41 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-22T05:47:41.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:41 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-22T05:47:41.546 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:41 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-22T05:47:41.546 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:41 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-22T05:47:41.546 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:41 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-22T05:47:41.546 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:41 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-22T05:47:41.546 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:41 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-22T05:47:41.546 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:41 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:47:41.546 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:41 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:47:41.546 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:41 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-22T05:47:41.546 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:41 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-22T05:47:41.546 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:41 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-22T05:47:41.546 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:41 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-22T05:47:41.546 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:41 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-22T05:47:41.546 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:41 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-22T05:47:41.546 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:41 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-22T05:47:41.547 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:41 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-22T05:47:41.547 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:41 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-22T05:47:41.547 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:41 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:47:41.547 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:41 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:47:41.547 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:41 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:47:41.547 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:41 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:47:41.547 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:41 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:47:41.547 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:41 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:47:41.547 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:41 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:47:41.547 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:41 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:47:41.547 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:41 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:47:41.547 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:41 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:47:41.547 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:41 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:47:41.547 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:41 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:47:41.547 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:41 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-22T05:47:41.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:41 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-22T05:47:41.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:41 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-22T05:47:41.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:41 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:47:42.442 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:42 smithi005 ceph-mon[84705]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:47:42.442 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:47:42.442 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:47:42.442 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:47:42.442 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:47:42.442 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:47:42.443 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:47:42.443 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:47:42.443 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:47:42.443 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:47:42.443 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:47:42.443 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:47:42.443 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:47:42.443 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:47:42.443 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:47:42.443 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:47:42.443 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:47:42.443 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:47:42.443 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:47:42.443 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:47:42.444 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:47:42.444 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:47:42.444 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:47:42.444 INFO:journalctl@ceph.mon.a.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:47:42.444 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:47:42.444 INFO:journalctl@ceph.mon.a.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:47:42.444 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:47:42.444 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:47:42.444 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:47:42.444 INFO:journalctl@ceph.mon.a.smithi005.stdout: return future.result(timeout) 2024-03-22T05:47:42.444 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:47:42.444 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.__get_result() 2024-03-22T05:47:42.444 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:47:42.444 INFO:journalctl@ceph.mon.a.smithi005.stdout: raise self._exception 2024-03-22T05:47:42.445 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:47:42.445 INFO:journalctl@ceph.mon.a.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:47:42.445 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:47:42.445 INFO:journalctl@ceph.mon.a.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:47:42.445 INFO:journalctl@ceph.mon.a.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:47:42.445 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:47:42.445 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:47:42.445 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:47:42.445 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:47:42.445 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:47:42.445 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:47:42.445 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:47:42.445 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:47:42.445 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:47:42.446 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:47:42.446 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:47:42.446 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:47:42.446 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:47:42.446 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:47:42.446 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:47:42.446 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:42 smithi005 ceph-mon[89904]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:47:42.447 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:47:42.447 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:47:42.447 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:47:42.447 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:47:42.447 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:47:42.447 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:47:42.447 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:47:42.447 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:47:42.447 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:47:42.447 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:47:42.447 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:47:42.447 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:47:42.447 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:47:42.447 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:47:42.448 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:47:42.448 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:47:42.448 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:47:42.448 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:47:42.448 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:47:42.448 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:47:42.448 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:47:42.448 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:47:42.448 INFO:journalctl@ceph.mon.c.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:47:42.448 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:47:42.448 INFO:journalctl@ceph.mon.c.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:47:42.448 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:47:42.448 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:47:42.449 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:47:42.449 INFO:journalctl@ceph.mon.c.smithi005.stdout: return future.result(timeout) 2024-03-22T05:47:42.449 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:47:42.449 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.__get_result() 2024-03-22T05:47:42.449 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:47:42.449 INFO:journalctl@ceph.mon.c.smithi005.stdout: raise self._exception 2024-03-22T05:47:42.449 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:47:42.449 INFO:journalctl@ceph.mon.c.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:47:42.449 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:47:42.449 INFO:journalctl@ceph.mon.c.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:47:42.449 INFO:journalctl@ceph.mon.c.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:47:42.449 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:47:42.449 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:47:42.449 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:47:42.449 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:47:42.450 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:47:42.450 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:47:42.450 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:47:42.450 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:47:42.450 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:47:42.450 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:47:42.450 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:47:42.450 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:47:42.450 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:47:42.450 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:47:42.450 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:47:42.450 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:47:42.450 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:47:42.451 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:47:42.451 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:47:42.451 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:47:42.451 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:42 smithi005 ceph-mon[89904]: Upgrade: Setting container_image for all nvmeof 2024-03-22T05:47:42.451 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:42 smithi005 ceph-mon[89904]: Upgrade: Finalizing container_image settings 2024-03-22T05:47:42.451 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:42 smithi005 ceph-mon[89904]: Upgrade: Complete! 2024-03-22T05:47:42.451 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:42 smithi005 ceph-mon[89904]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 877 B/s rd, 0 op/s 2024-03-22T05:47:42.451 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:42 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:47:42.451 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:42 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:47:42.451 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:42 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:47:42.451 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:42 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:47:42.451 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:42 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:47:42.451 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:42 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:47:42.452 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:47:42.452 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:47:42.452 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:47:42.452 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:47:42.452 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:47:42.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:42 smithi005 ceph-mon[84705]: Upgrade: Setting container_image for all nvmeof 2024-03-22T05:47:42.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:42 smithi005 ceph-mon[84705]: Upgrade: Finalizing container_image settings 2024-03-22T05:47:42.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:42 smithi005 ceph-mon[84705]: Upgrade: Complete! 2024-03-22T05:47:42.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:42 smithi005 ceph-mon[84705]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 877 B/s rd, 0 op/s 2024-03-22T05:47:42.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:42 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:47:42.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:42 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:47:42.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:42 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:47:42.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:42 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:47:42.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:42 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:47:42.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:42 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:47:42.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:42 smithi186 ceph-mon[67903]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:47:42.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:47:42.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:47:42.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:47:42.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:47:42.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:47:42.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:47:42.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:47:42.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:47:42.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:47:42.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:47:42.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:47:42.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:47:42.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:47:42.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:47:42.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:47:42.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:47:42.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:47:42.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:47:42.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:47:42.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:47:42.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:47:42.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:47:42.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:47:42.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:47:42.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:47:42.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:47:42.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:47:42.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:47:42.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: return future.result(timeout) 2024-03-22T05:47:42.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:47:42.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.__get_result() 2024-03-22T05:47:42.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:47:42.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: raise self._exception 2024-03-22T05:47:42.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:47:42.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:47:42.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:47:42.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:47:42.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:47:42.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:47:42.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:47:42.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:47:42.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:47:42.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:47:42.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:47:42.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:47:42.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:47:42.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:47:42.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:47:42.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:47:42.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:47:42.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:47:42.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:47:42.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:47:42.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:47:42.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:47:42.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:47:42.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:47:42.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:47:42.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:42 smithi186 ceph-mon[67903]: Upgrade: Setting container_image for all nvmeof 2024-03-22T05:47:42.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:42 smithi186 ceph-mon[67903]: Upgrade: Finalizing container_image settings 2024-03-22T05:47:42.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:42 smithi186 ceph-mon[67903]: Upgrade: Complete! 2024-03-22T05:47:42.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:42 smithi186 ceph-mon[67903]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 877 B/s rd, 0 op/s 2024-03-22T05:47:42.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:42 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:47:42.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:42 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:47:42.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:42 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:47:42.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:42 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:47:42.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:42 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:47:42.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:42 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:47:43.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:43 smithi005 ceph-mon[84705]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 968 B/s rd, 0 op/s 2024-03-22T05:47:43.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:43 smithi005 ceph-mon[84705]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 983 B/s rd, 0 op/s 2024-03-22T05:47:43.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:43 smithi005 ceph-mon[84705]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:47:43.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:43 smithi005 ceph-mon[84705]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:47:43.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:43 smithi005 ceph-mon[84705]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 40/723 objects degraded (5.533%), 8 pgs degraded) 2024-03-22T05:47:43.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:43 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:47:43.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:43 smithi005 ceph-mon[89904]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 968 B/s rd, 0 op/s 2024-03-22T05:47:43.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:43 smithi005 ceph-mon[89904]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 983 B/s rd, 0 op/s 2024-03-22T05:47:43.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:43 smithi005 ceph-mon[89904]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:47:43.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:43 smithi005 ceph-mon[89904]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:47:43.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:43 smithi005 ceph-mon[89904]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 40/723 objects degraded (5.533%), 8 pgs degraded) 2024-03-22T05:47:43.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:43 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:47:43.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:43 smithi186 ceph-mon[67903]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 968 B/s rd, 0 op/s 2024-03-22T05:47:43.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:43 smithi186 ceph-mon[67903]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 983 B/s rd, 0 op/s 2024-03-22T05:47:43.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:43 smithi186 ceph-mon[67903]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:47:43.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:43 smithi186 ceph-mon[67903]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:47:43.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:43 smithi186 ceph-mon[67903]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 40/723 objects degraded (5.533%), 8 pgs degraded) 2024-03-22T05:47:43.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:43 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:47:43.946 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph orch ps' 2024-03-22T05:47:44.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:44 smithi005 ceph-mon[89904]: from='client.54318 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:47:44.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:44 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:47:44.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:44 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:47:44.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:44 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:47:44.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:44 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:47:44.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:44 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:47:44.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:44 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:47:44.204 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:44 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:47:44.204 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:44 smithi005 ceph-mon[84705]: from='client.54318 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:47:44.204 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:44 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:47:44.204 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:44 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:47:44.205 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:44 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:47:44.205 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:44 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:47:44.205 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:44 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:47:44.205 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:44 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:47:44.205 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:44 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:47:44.206 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:47:44.024+0000 7f7f0ecc9700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:47:44.206 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:47:44.207 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:47:44.207 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:47:44.207 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:47:44.207 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:47:44.207 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:47:44.207 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:47:44.207 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:47:44.207 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:47:44.207 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:47:44.207 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:47:44.207 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:47:44.207 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:47:44.207 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:47:44.207 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:47:44.208 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:47:44.208 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:47:44.208 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:47:44.208 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:47:44.208 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:47:44.208 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:47:44.208 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:47:44.208 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:47:44.208 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:47:44.208 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:47:44.208 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:47:44.208 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:47:44.208 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:47:44.208 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return future.result(timeout) 2024-03-22T05:47:44.208 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:47:44.209 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.__get_result() 2024-03-22T05:47:44.209 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:47:44.209 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: raise self._exception 2024-03-22T05:47:44.209 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:47:44.209 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: config_blobs=daemon_spec.final_config, 2024-03-22T05:47:44.209 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:47:44.209 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:47:44.209 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:47:44.209 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:47:44.209 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:47:44.209 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:47:44.209 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:47:44.209 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:47:44.209 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:47:44.209 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:47:44.210 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:47:44.210 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:47:44.210 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:47:44.210 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:47:44.210 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:47:44.210 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:47:44.210 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:47:44.210 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:47:44.210 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:47:44.210 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:47:44.210 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:47:44.210 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:47:44.210 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:44 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:47:44.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:44 smithi186 ceph-mon[67903]: from='client.54318 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:47:44.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:44 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:47:44.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:44 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:47:44.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:44 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:47:44.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:44 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:47:44.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:44 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:47:44.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:44 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:47:44.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:44 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:47:45.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:45 smithi005 ceph-mon[84705]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:47:45.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:45 smithi005 ceph-mon[84705]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 1 op/s 2024-03-22T05:47:45.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:45 smithi005 ceph-mon[84705]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:47:45.203 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:47:45.203 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:47:45.203 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:47:45.203 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:47:45.203 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:47:45.203 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:47:45.204 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:47:45.204 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:47:45.204 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:47:45.204 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:47:45.204 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:47:45.204 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:47:45.204 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:47:45.204 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:47:45.204 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:47:45.204 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:47:45.204 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:47:45.205 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:47:45.205 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:47:45.205 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:47:45.205 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:47:45.205 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:47:45.205 INFO:journalctl@ceph.mon.a.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:47:45.205 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:47:45.205 INFO:journalctl@ceph.mon.a.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:47:45.205 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:47:45.205 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:47:45.205 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:47:45.205 INFO:journalctl@ceph.mon.a.smithi005.stdout: return future.result(timeout) 2024-03-22T05:47:45.206 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:47:45.206 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.__get_result() 2024-03-22T05:47:45.206 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:47:45.206 INFO:journalctl@ceph.mon.a.smithi005.stdout: raise self._exception 2024-03-22T05:47:45.206 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:47:45.206 INFO:journalctl@ceph.mon.a.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:47:45.206 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:47:45.206 INFO:journalctl@ceph.mon.a.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:47:45.206 INFO:journalctl@ceph.mon.a.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:47:45.206 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:47:45.206 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:47:45.207 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:47:45.207 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:47:45.207 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:47:45.207 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:47:45.207 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:47:45.207 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:47:45.207 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:47:45.207 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:47:45.207 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:47:45.207 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:47:45.207 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:47:45.208 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:47:45.208 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:47:45.208 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:47:45.208 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:47:45.208 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:47:45.208 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:47:45.208 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:47:45.208 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:45 smithi005 ceph-mon[84705]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-22T05:47:45.208 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:45 smithi005 ceph-mon[84705]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-22T05:47:45.208 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:45 smithi005 ceph-mon[84705]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:47:45.208 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:45 smithi005 ceph-mon[84705]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:47:45.209 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:45 smithi005 ceph-mon[89904]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:47:45.209 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:45 smithi005 ceph-mon[89904]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 1 op/s 2024-03-22T05:47:45.209 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:45 smithi005 ceph-mon[89904]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:47:45.209 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:47:45.209 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:47:45.209 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:47:45.209 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:47:45.210 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:47:45.210 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:47:45.210 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:47:45.210 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:47:45.210 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:47:45.210 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:47:45.210 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:47:45.210 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:47:45.210 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:47:45.210 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:47:45.210 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:47:45.210 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:47:45.211 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:47:45.211 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:47:45.211 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:47:45.211 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:47:45.211 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:47:45.211 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:47:45.211 INFO:journalctl@ceph.mon.c.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:47:45.211 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:47:45.211 INFO:journalctl@ceph.mon.c.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:47:45.211 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:47:45.211 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:47:45.212 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:47:45.212 INFO:journalctl@ceph.mon.c.smithi005.stdout: return future.result(timeout) 2024-03-22T05:47:45.212 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:47:45.212 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.__get_result() 2024-03-22T05:47:45.212 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:47:45.212 INFO:journalctl@ceph.mon.c.smithi005.stdout: raise self._exception 2024-03-22T05:47:45.212 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:47:45.212 INFO:journalctl@ceph.mon.c.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:47:45.212 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:47:45.213 INFO:journalctl@ceph.mon.c.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:47:45.213 INFO:journalctl@ceph.mon.c.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:47:45.213 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:47:45.213 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:47:45.213 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:47:45.213 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:47:45.213 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:47:45.213 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:47:45.213 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:47:45.213 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:47:45.213 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:47:45.213 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:47:45.214 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:47:45.214 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:47:45.214 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:47:45.214 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:47:45.214 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:47:45.214 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:47:45.214 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:47:45.214 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:47:45.214 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:47:45.214 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:47:45.214 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:45 smithi005 ceph-mon[89904]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-22T05:47:45.215 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:45 smithi005 ceph-mon[89904]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-22T05:47:45.215 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:45 smithi005 ceph-mon[89904]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:47:45.215 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:45 smithi005 ceph-mon[89904]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:47:45.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:45 smithi186 ceph-mon[67903]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:47:45.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:45 smithi186 ceph-mon[67903]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 1 op/s 2024-03-22T05:47:45.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:45 smithi186 ceph-mon[67903]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:47:45.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:47:45.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:47:45.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:47:45.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:47:45.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:47:45.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:47:45.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:47:45.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:47:45.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:47:45.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:47:45.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:47:45.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:47:45.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:47:45.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:47:45.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:47:45.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:47:45.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:47:45.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:47:45.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:47:45.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:47:45.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:47:45.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:47:45.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:47:45.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:47:45.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:47:45.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:47:45.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:47:45.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:47:45.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: return future.result(timeout) 2024-03-22T05:47:45.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:47:45.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.__get_result() 2024-03-22T05:47:45.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:47:45.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: raise self._exception 2024-03-22T05:47:45.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:47:45.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:47:45.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:47:45.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:47:45.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:47:45.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:47:45.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:47:45.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:47:45.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:47:45.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:47:45.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:47:45.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:47:45.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:47:45.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:47:45.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:47:45.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:47:45.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:47:45.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:47:45.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:47:45.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:47:45.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:47:45.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:47:45.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:47:45.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:47:45.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:47:45.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:45 smithi186 ceph-mon[67903]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-22T05:47:45.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:45 smithi186 ceph-mon[67903]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-22T05:47:45.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:45 smithi186 ceph-mon[67903]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:47:45.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:45 smithi186 ceph-mon[67903]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:47:45.934 INFO:teuthology.orchestra.run.smithi005.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T05:47:45.934 INFO:teuthology.orchestra.run.smithi005.stdout:alertmanager.a smithi005 *:9093,9094 running (5m) 11s ago 27m 28.0M - 0.25.0 c8568f914cd2 ee99dfc435e8 2024-03-22T05:47:45.934 INFO:teuthology.orchestra.run.smithi005.stdout:grafana.a smithi186 *:3000 running (5m) 2m ago 26m 81.5M - 9.4.12 2bacad6d85d8 16fd57dbbd71 2024-03-22T05:47:45.934 INFO:teuthology.orchestra.run.smithi005.stdout:iscsi.foo.smithi005.mcdwbp smithi005 unknown 11s ago 26m 64.6M - 3.5 e1d6a67b021e 5215e03bf81b 2024-03-22T05:47:45.934 INFO:teuthology.orchestra.run.smithi005.stdout:mgr.x smithi186 *:8443,9283,8765 running (6m) 2m ago 31m 448M - 19.0.0-2299-gea01217f aa461e7c5e7f c556421a73f3 2024-03-22T05:47:45.934 INFO:teuthology.orchestra.run.smithi005.stdout:mgr.y smithi005 *:8443,9283,8765 running (16m) 11s ago 33m 507M - 19.0.0-2299-gea01217f aa461e7c5e7f 62a813eebc0d 2024-03-22T05:47:45.934 INFO:teuthology.orchestra.run.smithi005.stdout:mon.a smithi005 running (3m) 11s ago 33m 50.2M 2048M 19.0.0-2299-gea01217f aa461e7c5e7f 111a2aa24dd9 2024-03-22T05:47:45.935 INFO:teuthology.orchestra.run.smithi005.stdout:mon.b smithi186 running (3m) 2m ago 31m 39.1M 2048M 19.0.0-2299-gea01217f aa461e7c5e7f c1798feac15b 2024-03-22T05:47:45.935 INFO:teuthology.orchestra.run.smithi005.stdout:mon.c smithi005 running (2m) 11s ago 32m 37.8M 2048M 19.0.0-2299-gea01217f aa461e7c5e7f 708c72ff74f5 2024-03-22T05:47:45.935 INFO:teuthology.orchestra.run.smithi005.stdout:node-exporter.a smithi005 *:9100 running (6m) 11s ago 27m 17.3M - 1.5.0 0da6a335fe13 f635cd72b9d3 2024-03-22T05:47:45.935 INFO:teuthology.orchestra.run.smithi005.stdout:node-exporter.b smithi186 *:9100 running (5m) 2m ago 27m 15.1M - 1.5.0 0da6a335fe13 d7e0b3d20191 2024-03-22T05:47:45.935 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0 smithi005 running (53s) 11s ago 31m 42.6M 2993M 19.0.0-2299-gea01217f aa461e7c5e7f 31b6cdee5f5b 2024-03-22T05:47:45.935 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1 smithi005 running (16s) 11s ago 30m 12.4M 2993M 19.0.0-2299-gea01217f aa461e7c5e7f 7d21bc9fd3c2 2024-03-22T05:47:45.935 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2 smithi005 running (30m) 11s ago 30m 51.4M 2993M 17.2.0 e1d6a67b021e 462ea63ad0e5 2024-03-22T05:47:45.935 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3 smithi005 running (30m) 11s ago 30m 58.4M 2993M 17.2.0 e1d6a67b021e b25432a7239b 2024-03-22T05:47:45.935 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4 smithi186 running (29m) 2m ago 29m 56.4M 3248M 17.2.0 e1d6a67b021e cd963671f340 2024-03-22T05:47:45.935 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5 smithi186 running (29m) 2m ago 29m 51.7M 3248M 17.2.0 e1d6a67b021e f98c87f1a2cd 2024-03-22T05:47:45.935 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6 smithi186 running (29m) 2m ago 29m 52.0M 3248M 17.2.0 e1d6a67b021e 704ac49c68c4 2024-03-22T05:47:45.935 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7 smithi186 running (28m) 2m ago 28m 54.3M 3248M 17.2.0 e1d6a67b021e f415b90edf63 2024-03-22T05:47:45.936 INFO:teuthology.orchestra.run.smithi005.stdout:prometheus.a smithi186 *:9095 running (5m) 2m ago 27m 67.2M - 2.43.0 a07b618ecd1d a5a043588d6d 2024-03-22T05:47:45.936 INFO:teuthology.orchestra.run.smithi005.stdout:rgw.foo.smithi005.exdhqu smithi005 *:8000 running (26m) 11s ago 26m 81.0M - 17.2.0 e1d6a67b021e 6bb27fe11553 2024-03-22T05:47:45.936 INFO:teuthology.orchestra.run.smithi005.stdout:rgw.foo.smithi186.mcgodk smithi186 *:8000 running (26m) 2m ago 26m 78.2M - 17.2.0 e1d6a67b021e 19b5c20a34f7 2024-03-22T05:47:46.571 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph versions | jq -e '"'"'.osd | length == 2'"'"'' 2024-03-22T05:47:47.183 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:47 smithi005 ceph-mon[84705]: from='client.54324 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:47:47.183 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:47 smithi005 ceph-mon[84705]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-22T05:47:47.183 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:46 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:47:46] "GET /metrics HTTP/1.1" 200 38243 "" "Prometheus/2.43.0" 2024-03-22T05:47:47.183 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:47 smithi005 ceph-mon[89904]: from='client.54324 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:47:47.183 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:47 smithi005 ceph-mon[89904]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-22T05:47:47.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:47 smithi186 ceph-mon[67903]: from='client.54324 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:47:47.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:47 smithi186 ceph-mon[67903]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-22T05:47:48.324 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:47:48.064+0000 7f7f0ecc9700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:47:48.324 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:47:48.324 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:47:48.324 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:47:48.324 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:47:48.324 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:47:48.325 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:47:48.325 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:47:48.325 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:47:48.325 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:47:48.325 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:47:48.325 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:47:48.325 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:47:48.325 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:47:48.325 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:47:48.325 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:47:48.325 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:47:48.325 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:47:48.325 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:47:48.326 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:47:48.326 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:47:48.326 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:47:48.326 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:47:48.326 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:47:48.326 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:47:48.326 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:47:48.326 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:47:48.326 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:47:48.327 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:47:48.327 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return future.result(timeout) 2024-03-22T05:47:48.327 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:47:48.327 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.__get_result() 2024-03-22T05:47:48.327 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:47:48.327 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: raise self._exception 2024-03-22T05:47:48.327 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:47:48.327 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: config_blobs=daemon_spec.final_config, 2024-03-22T05:47:48.327 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:47:48.327 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:47:48.327 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:47:48.327 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:47:48.327 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:47:48.327 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:47:48.327 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:47:48.328 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:47:48.328 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:47:48.328 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:47:48.328 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:47:48.328 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:47:48.328 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:47:48.328 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:47:48.328 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:47:48.328 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:47:48.328 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:47:48.328 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:47:48.328 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:47:48.328 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:47:48.328 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:47:48.328 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:47:48.329 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:47:48.586 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:48 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:47:48.586 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:48 smithi005 ceph-mon[84705]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 795 B/s rd, 0 op/s 2024-03-22T05:47:48.586 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:48 smithi005 ceph-mon[84705]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:47:48.586 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:47:48.587 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:47:48.587 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:47:48.587 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:47:48.587 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:47:48.587 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:47:48.587 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:47:48.587 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:47:48.587 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:47:48.587 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:47:48.587 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:47:48.587 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:47:48.587 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:47:48.587 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:47:48.587 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:47:48.587 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:47:48.588 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:47:48.588 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:47:48.588 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:47:48.588 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:47:48.588 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:47:48.588 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:47:48.588 INFO:journalctl@ceph.mon.a.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:47:48.588 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:47:48.588 INFO:journalctl@ceph.mon.a.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:47:48.588 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:47:48.588 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:47:48.588 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:47:48.588 INFO:journalctl@ceph.mon.a.smithi005.stdout: return future.result(timeout) 2024-03-22T05:47:48.588 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:47:48.588 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.__get_result() 2024-03-22T05:47:48.589 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:47:48.589 INFO:journalctl@ceph.mon.a.smithi005.stdout: raise self._exception 2024-03-22T05:47:48.589 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:47:48.589 INFO:journalctl@ceph.mon.a.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:47:48.589 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:47:48.589 INFO:journalctl@ceph.mon.a.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:47:48.589 INFO:journalctl@ceph.mon.a.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:47:48.589 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:47:48.589 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:47:48.589 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:47:48.589 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:47:48.589 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:47:48.589 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:47:48.589 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:47:48.589 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:47:48.590 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:47:48.590 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:47:48.590 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:47:48.590 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:47:48.590 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:47:48.590 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:47:48.590 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:47:48.590 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:47:48.590 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:47:48.590 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:47:48.590 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:47:48.590 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:47:48.591 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:48 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:47:48.591 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:48 smithi005 ceph-mon[89904]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 795 B/s rd, 0 op/s 2024-03-22T05:47:48.591 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:48 smithi005 ceph-mon[89904]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:47:48.591 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:47:48.591 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:47:48.591 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:47:48.591 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:47:48.591 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:47:48.591 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:47:48.591 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:47:48.591 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:47:48.591 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:47:48.591 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:47:48.592 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:47:48.592 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:47:48.592 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:47:48.592 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:47:48.592 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:47:48.592 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:47:48.592 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:47:48.592 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:47:48.592 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:47:48.592 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:47:48.592 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:47:48.592 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:47:48.592 INFO:journalctl@ceph.mon.c.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:47:48.592 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:47:48.593 INFO:journalctl@ceph.mon.c.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:47:48.593 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:47:48.593 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:47:48.593 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:47:48.593 INFO:journalctl@ceph.mon.c.smithi005.stdout: return future.result(timeout) 2024-03-22T05:47:48.593 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:47:48.593 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.__get_result() 2024-03-22T05:47:48.593 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:47:48.593 INFO:journalctl@ceph.mon.c.smithi005.stdout: raise self._exception 2024-03-22T05:47:48.593 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:47:48.593 INFO:journalctl@ceph.mon.c.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:47:48.593 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:47:48.593 INFO:journalctl@ceph.mon.c.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:47:48.594 INFO:journalctl@ceph.mon.c.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:47:48.594 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:47:48.594 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:47:48.594 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:47:48.594 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:47:48.594 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:47:48.594 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:47:48.594 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:47:48.594 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:47:48.594 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:47:48.595 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:47:48.595 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:47:48.595 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:47:48.595 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:47:48.595 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:47:48.595 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:47:48.595 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:47:48.595 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:47:48.595 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:47:48.595 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:47:48.595 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:47:49.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:48 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:47:49.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:48 smithi186 ceph-mon[67903]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 795 B/s rd, 0 op/s 2024-03-22T05:47:49.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:48 smithi186 ceph-mon[67903]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:47:49.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:47:49.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:47:49.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:47:49.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:47:49.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:47:49.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:47:49.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:47:49.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:47:49.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:47:49.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:47:49.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:47:49.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:47:49.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:47:49.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:47:49.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:47:49.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:47:49.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:47:49.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:47:49.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:47:49.048 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:47:49.048 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:47:49.048 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:47:49.048 INFO:journalctl@ceph.mon.b.smithi186.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:47:49.048 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:47:49.048 INFO:journalctl@ceph.mon.b.smithi186.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:47:49.048 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:47:49.049 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:47:49.049 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:47:49.049 INFO:journalctl@ceph.mon.b.smithi186.stdout: return future.result(timeout) 2024-03-22T05:47:49.049 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:47:49.049 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.__get_result() 2024-03-22T05:47:49.049 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:47:49.049 INFO:journalctl@ceph.mon.b.smithi186.stdout: raise self._exception 2024-03-22T05:47:49.049 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:47:49.050 INFO:journalctl@ceph.mon.b.smithi186.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:47:49.050 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:47:49.050 INFO:journalctl@ceph.mon.b.smithi186.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:47:49.050 INFO:journalctl@ceph.mon.b.smithi186.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:47:49.050 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:47:49.050 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:47:49.050 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:47:49.050 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:47:49.051 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:47:49.051 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:47:49.051 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:47:49.051 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:47:49.051 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:47:49.051 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:47:49.052 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:47:49.052 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:47:49.052 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:47:49.052 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:47:49.052 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:47:49.052 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:47:49.053 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:47:49.053 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:47:49.053 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:47:49.053 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:47:49.151 INFO:teuthology.orchestra.run.smithi005.stdout:true 2024-03-22T05:47:49.772 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:49 smithi005 ceph-mon[84705]: from='client.? 172.21.15.5:0/1333773500' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:47:49.773 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:49 smithi005 ceph-mon[89904]: from='client.? 172.21.15.5:0/1333773500' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:47:49.795 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '"'"'.up_to_date | length == 7'"'"'' 2024-03-22T05:47:50.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:49 smithi186 ceph-mon[67903]: from='client.? 172.21.15.5:0/1333773500' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:47:50.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:50 smithi005 ceph-mon[84705]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:47:50.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:50 smithi005 ceph-mon[89904]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:47:51.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:50 smithi186 ceph-mon[67903]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:47:53.342 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:53 smithi005 ceph-mon[89904]: from='client.54336 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:47:53.342 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:53 smithi005 ceph-mon[89904]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 606 B/s rd, 0 op/s 2024-03-22T05:47:53.342 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:53 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:47:53.342 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:53 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:47:53.343 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:53 smithi005 ceph-mon[84705]: from='client.54336 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:47:53.343 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:53 smithi005 ceph-mon[84705]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 606 B/s rd, 0 op/s 2024-03-22T05:47:53.343 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:53 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:47:53.343 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:53 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:47:53.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:53 smithi186 ceph-mon[67903]: from='client.54336 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:47:53.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:53 smithi186 ceph-mon[67903]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 606 B/s rd, 0 op/s 2024-03-22T05:47:53.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:53 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:47:53.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:53 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:47:54.149 INFO:teuthology.orchestra.run.smithi005.stdout:true 2024-03-22T05:47:54.774 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph orch upgrade status' 2024-03-22T05:47:55.418 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:55 smithi005 ceph-mon[84705]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:47:55.419 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:55 smithi005 ceph-mon[84705]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:47:55.419 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:55 smithi005 ceph-mon[89904]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:47:55.419 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:55 smithi005 ceph-mon[89904]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:47:55.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:55 smithi186 ceph-mon[67903]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:47:55.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:55 smithi186 ceph-mon[67903]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:47:56.853 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-03-22T05:47:56.853 INFO:teuthology.orchestra.run.smithi005.stdout: "target_image": null, 2024-03-22T05:47:56.853 INFO:teuthology.orchestra.run.smithi005.stdout: "in_progress": false, 2024-03-22T05:47:56.853 INFO:teuthology.orchestra.run.smithi005.stdout: "which": "", 2024-03-22T05:47:56.853 INFO:teuthology.orchestra.run.smithi005.stdout: "services_complete": [], 2024-03-22T05:47:56.853 INFO:teuthology.orchestra.run.smithi005.stdout: "progress": null, 2024-03-22T05:47:56.854 INFO:teuthology.orchestra.run.smithi005.stdout: "message": "", 2024-03-22T05:47:56.854 INFO:teuthology.orchestra.run.smithi005.stdout: "is_paused": false 2024-03-22T05:47:56.854 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-03-22T05:47:57.230 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:57 smithi005 ceph-mon[84705]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:47:57.231 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:47:56 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:47:56] "GET /metrics HTTP/1.1" 200 38232 "" "Prometheus/2.43.0" 2024-03-22T05:47:57.231 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:57 smithi005 ceph-mon[89904]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:47:57.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:57 smithi186 ceph-mon[67903]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:47:57.633 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph health detail' 2024-03-22T05:47:58.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:58 smithi005 ceph-mon[84705]: from='client.54342 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:47:58.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:58 smithi005 ceph-mon[89904]: from='client.54342 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:47:58.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:58 smithi186 ceph-mon[67903]: from='client.54342 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:47:59.371 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:47:59 smithi005 ceph-mon[84705]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:47:59.371 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:47:59 smithi005 ceph-mon[89904]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:47:59.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:47:59 smithi186 ceph-mon[67903]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:47:59.873 INFO:teuthology.orchestra.run.smithi005.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-22T05:47:59.873 INFO:teuthology.orchestra.run.smithi005.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-22T05:47:59.873 INFO:teuthology.orchestra.run.smithi005.stdout: daemon iscsi.foo.smithi005.mcdwbp on smithi005 is in unknown state 2024-03-22T05:48:00.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:00 smithi005 ceph-mon[84705]: from='client.? 172.21.15.5:0/501634720' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T05:48:00.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:00 smithi005 ceph-mon[89904]: from='client.? 172.21.15.5:0/501634720' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T05:48:00.445 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:00 smithi186 ceph-mon[67903]: from='client.? 172.21.15.5:0/501634720' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T05:48:00.520 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types crash,osd --limit 1' 2024-03-22T05:48:01.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:01 smithi005 ceph-mon[84705]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:48:01.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:01 smithi005 ceph-mon[89904]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:48:01.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:01 smithi186 ceph-mon[67903]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:48:03.393 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:03 smithi005 ceph-mon[84705]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:48:03.394 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:03 smithi005 ceph-mon[89904]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:48:03.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:03 smithi186 ceph-mon[67903]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:48:04.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:04 smithi005 ceph-mon[84705]: from='client.54354 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37", "daemon_types": "crash,osd", "limit": 1, "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:48:04.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:04 smithi005 ceph-mon[89904]: from='client.54354 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37", "daemon_types": "crash,osd", "limit": 1, "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:48:04.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:04 smithi186 ceph-mon[67903]: from='client.54354 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37", "daemon_types": "crash,osd", "limit": 1, "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:48:04.645 INFO:teuthology.orchestra.run.smithi005.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37 2024-03-22T05:48:05.254 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:05 smithi005 ceph-mon[84705]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:48:05.254 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:05 smithi005 ceph-mon[84705]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:48:05.254 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:05 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:48:05.255 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:05 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:48:05.255 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:05 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:48:05.255 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:05 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:48:05.255 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:05 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:48:05.255 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:05 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:48:05.255 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:05 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:48:05.255 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:05 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:48:05.255 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:05 smithi005 ceph-mon[89904]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:48:05.255 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:05 smithi005 ceph-mon[89904]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:48:05.255 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:05 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:48:05.255 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:05 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:48:05.256 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:05 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:48:05.256 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:05 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:48:05.256 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:05 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:48:05.256 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:05 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:48:05.256 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:05 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:48:05.256 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:05 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:48:05.277 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-03-22T05:48:05.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:05 smithi186 ceph-mon[67903]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:48:05.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:05 smithi186 ceph-mon[67903]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:48:05.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:05 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:48:05.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:05 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:48:05.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:05 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:48:05.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:05 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:48:05.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:05 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:48:05.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:05 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:48:05.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:05 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:48:05.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:05 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:48:06.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:06 smithi005 ceph-mon[89904]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37 2024-03-22T05:48:06.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:06 smithi005 ceph-mon[89904]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 962 B/s rd, 0 op/s 2024-03-22T05:48:06.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:06 smithi005 ceph-mon[89904]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:48:06.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:06 smithi005 ceph-mon[89904]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:48:06.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:06 smithi005 ceph-mon[89904]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:48:06.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:06 smithi005 ceph-mon[84705]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37 2024-03-22T05:48:06.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:06 smithi005 ceph-mon[84705]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 962 B/s rd, 0 op/s 2024-03-22T05:48:06.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:06 smithi005 ceph-mon[84705]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:48:06.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:06 smithi005 ceph-mon[84705]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:48:06.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:06 smithi005 ceph-mon[84705]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:48:06.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:06 smithi186 ceph-mon[67903]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37 2024-03-22T05:48:06.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:06 smithi186 ceph-mon[67903]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 962 B/s rd, 0 op/s 2024-03-22T05:48:06.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:06 smithi186 ceph-mon[67903]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:48:06.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:06 smithi186 ceph-mon[67903]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:48:06.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:06 smithi186 ceph-mon[67903]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:48:07.452 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:48:06] "GET /metrics HTTP/1.1" 200 38232 "" "Prometheus/2.43.0" 2024-03-22T05:48:07.453 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:07 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:48:07.146+0000 7f7f0ecc9700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:48:07.453 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:07 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:48:07.453 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:07 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:48:07.453 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:07 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:48:07.453 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:07 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:48:07.453 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:07 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:48:07.453 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:07 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:48:07.453 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:07 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:48:07.453 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:07 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:48:07.453 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:07 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:48:07.453 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:07 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:48:07.453 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:07 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:48:07.454 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:07 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:48:07.454 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:07 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:48:07.454 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:07 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:48:07.454 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:07 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:48:07.454 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:07 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:48:07.454 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:07 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:48:07.457 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:07 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:48:07.457 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:07 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:48:07.457 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:07 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:48:07.457 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:07 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:48:07.457 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:07 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:48:07.458 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:07 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:48:07.458 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:07 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:48:07.458 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:07 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:48:07.458 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:07 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:48:07.458 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:07 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:48:07.458 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:07 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:48:07.458 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:07 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return future.result(timeout) 2024-03-22T05:48:07.458 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:07 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:48:07.458 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:07 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.__get_result() 2024-03-22T05:48:07.458 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:07 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:48:07.458 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:07 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: raise self._exception 2024-03-22T05:48:07.458 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:07 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:48:07.458 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:07 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: config_blobs=daemon_spec.final_config, 2024-03-22T05:48:07.459 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:07 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:48:07.459 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:07 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:48:07.459 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:07 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:48:07.459 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:07 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:48:07.459 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:07 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:48:07.459 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:07 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:48:07.459 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:07 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:48:07.459 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:07 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:48:07.459 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:07 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:48:07.459 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:07 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:48:07.459 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:07 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:48:07.459 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:07 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:48:07.459 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:07 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:48:07.459 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:07 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:48:07.460 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:07 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:48:07.460 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:07 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:48:07.460 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:07 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:48:07.460 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:07 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:48:07.460 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:07 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:48:07.460 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:07 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:48:07.460 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:07 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:48:07.460 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:07 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:48:07.460 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:07 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:48:08.231 INFO:teuthology.orchestra.run.smithi005.stdout:true 2024-03-22T05:48:08.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:08 smithi005 ceph-mon[84705]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:48:08.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:08 smithi005 ceph-mon[84705]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:48:08.453 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:48:08.453 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:48:08.453 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:48:08.453 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:48:08.453 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:48:08.453 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:48:08.453 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:48:08.453 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:48:08.453 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:48:08.454 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:48:08.454 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:48:08.454 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:48:08.454 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:48:08.454 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:48:08.454 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:48:08.454 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:48:08.454 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:48:08.454 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:48:08.454 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:48:08.454 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:48:08.454 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:48:08.454 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:48:08.454 INFO:journalctl@ceph.mon.a.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:48:08.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:48:08.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:48:08.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:48:08.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:48:08.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:48:08.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: return future.result(timeout) 2024-03-22T05:48:08.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:48:08.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.__get_result() 2024-03-22T05:48:08.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:48:08.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: raise self._exception 2024-03-22T05:48:08.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:48:08.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:48:08.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:48:08.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:48:08.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:48:08.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:48:08.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:48:08.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:48:08.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:48:08.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:48:08.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:48:08.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:48:08.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:48:08.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:48:08.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:48:08.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:48:08.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:48:08.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:48:08.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:48:08.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:48:08.457 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:08 smithi005 ceph-mon[89904]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:48:08.457 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:08 smithi005 ceph-mon[89904]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:48:08.457 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:48:08.457 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:48:08.457 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:48:08.457 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:48:08.457 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:48:08.457 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:48:08.457 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:48:08.457 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:48:08.458 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:48:08.458 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:48:08.458 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:48:08.458 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:48:08.458 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:48:08.458 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:48:08.458 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:48:08.458 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:48:08.458 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:48:08.458 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:48:08.458 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:48:08.458 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:48:08.458 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:48:08.458 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:48:08.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:48:08.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:48:08.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:48:08.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:48:08.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:48:08.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:48:08.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: return future.result(timeout) 2024-03-22T05:48:08.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:48:08.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.__get_result() 2024-03-22T05:48:08.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:48:08.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: raise self._exception 2024-03-22T05:48:08.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:48:08.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:48:08.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:48:08.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:48:08.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:48:08.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:48:08.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:48:08.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:48:08.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:48:08.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:48:08.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:48:08.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:48:08.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:48:08.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:48:08.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:48:08.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:48:08.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:48:08.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:48:08.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:48:08.461 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:48:08.461 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:48:08.461 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:48:08.461 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:48:08.461 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:48:08.461 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:48:08.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:08 smithi005 ceph-mon[84705]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37 2024-03-22T05:48:08.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:08 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:48:08.461 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:48:08.462 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:48:08.462 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:48:08.462 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:48:08.462 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:48:08.462 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:08 smithi005 ceph-mon[89904]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37 2024-03-22T05:48:08.462 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:08 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:48:08.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:08 smithi186 ceph-mon[67903]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:48:08.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:08 smithi186 ceph-mon[67903]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:48:08.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:48:08.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:48:08.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:48:08.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:48:08.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:48:08.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:48:08.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:48:08.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:48:08.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:48:08.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:48:08.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:48:08.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:48:08.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:48:08.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:48:08.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:48:08.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:48:08.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:48:08.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:48:08.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:48:08.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:48:08.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:48:08.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:48:08.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:48:08.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:48:08.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:48:08.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:48:08.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:48:08.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:48:08.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: return future.result(timeout) 2024-03-22T05:48:08.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:48:08.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.__get_result() 2024-03-22T05:48:08.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:48:08.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: raise self._exception 2024-03-22T05:48:08.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:48:08.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:48:08.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:48:08.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:48:08.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:48:08.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:48:08.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:48:08.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:48:08.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:48:08.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:48:08.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:48:08.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:48:08.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:48:08.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:48:08.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:48:08.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:48:08.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:48:08.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:48:08.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:48:08.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:48:08.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:48:08.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:48:08.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:48:08.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:48:08.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:48:08.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:08 smithi186 ceph-mon[67903]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37 2024-03-22T05:48:08.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:08 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:48:08.933 INFO:teuthology.orchestra.run.smithi005.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T05:48:08.934 INFO:teuthology.orchestra.run.smithi005.stdout:alertmanager.a smithi005 *:9093,9094 running (5m) 34s ago 27m 28.0M - 0.25.0 c8568f914cd2 ee99dfc435e8 2024-03-22T05:48:08.934 INFO:teuthology.orchestra.run.smithi005.stdout:grafana.a smithi186 *:3000 running (5m) 3m ago 27m 81.5M - 9.4.12 2bacad6d85d8 16fd57dbbd71 2024-03-22T05:48:08.934 INFO:teuthology.orchestra.run.smithi005.stdout:iscsi.foo.smithi005.mcdwbp smithi005 unknown 34s ago 26m 64.6M - 3.5 e1d6a67b021e 5215e03bf81b 2024-03-22T05:48:08.934 INFO:teuthology.orchestra.run.smithi005.stdout:mgr.x smithi186 *:8443,9283,8765 running (6m) 3m ago 32m 448M - 19.0.0-2299-gea01217f aa461e7c5e7f c556421a73f3 2024-03-22T05:48:08.934 INFO:teuthology.orchestra.run.smithi005.stdout:mgr.y smithi005 *:8443,9283,8765 running (16m) 34s ago 33m 507M - 19.0.0-2299-gea01217f aa461e7c5e7f 62a813eebc0d 2024-03-22T05:48:08.934 INFO:teuthology.orchestra.run.smithi005.stdout:mon.a smithi005 running (3m) 34s ago 33m 50.2M 2048M 19.0.0-2299-gea01217f aa461e7c5e7f 111a2aa24dd9 2024-03-22T05:48:08.934 INFO:teuthology.orchestra.run.smithi005.stdout:mon.b smithi186 running (4m) 3m ago 32m 39.1M 2048M 19.0.0-2299-gea01217f aa461e7c5e7f c1798feac15b 2024-03-22T05:48:08.934 INFO:teuthology.orchestra.run.smithi005.stdout:mon.c smithi005 running (2m) 34s ago 32m 37.8M 2048M 19.0.0-2299-gea01217f aa461e7c5e7f 708c72ff74f5 2024-03-22T05:48:08.934 INFO:teuthology.orchestra.run.smithi005.stdout:node-exporter.a smithi005 *:9100 running (6m) 34s ago 28m 17.3M - 1.5.0 0da6a335fe13 f635cd72b9d3 2024-03-22T05:48:08.934 INFO:teuthology.orchestra.run.smithi005.stdout:node-exporter.b smithi186 *:9100 running (6m) 3m ago 28m 15.1M - 1.5.0 0da6a335fe13 d7e0b3d20191 2024-03-22T05:48:08.934 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0 smithi005 running (76s) 34s ago 31m 42.6M 2993M 19.0.0-2299-gea01217f aa461e7c5e7f 31b6cdee5f5b 2024-03-22T05:48:08.934 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1 smithi005 running (39s) 34s ago 31m 12.4M 2993M 19.0.0-2299-gea01217f aa461e7c5e7f 7d21bc9fd3c2 2024-03-22T05:48:08.935 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2 smithi005 running (30m) 34s ago 30m 51.4M 2993M 17.2.0 e1d6a67b021e 462ea63ad0e5 2024-03-22T05:48:08.935 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3 smithi005 running (30m) 34s ago 30m 58.4M 2993M 17.2.0 e1d6a67b021e b25432a7239b 2024-03-22T05:48:08.935 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4 smithi186 running (30m) 3m ago 30m 56.4M 3248M 17.2.0 e1d6a67b021e cd963671f340 2024-03-22T05:48:08.935 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5 smithi186 running (29m) 3m ago 29m 51.7M 3248M 17.2.0 e1d6a67b021e f98c87f1a2cd 2024-03-22T05:48:08.935 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6 smithi186 running (29m) 3m ago 29m 52.0M 3248M 17.2.0 e1d6a67b021e 704ac49c68c4 2024-03-22T05:48:08.935 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7 smithi186 running (29m) 3m ago 29m 54.3M 3248M 17.2.0 e1d6a67b021e f415b90edf63 2024-03-22T05:48:08.935 INFO:teuthology.orchestra.run.smithi005.stdout:prometheus.a smithi186 *:9095 running (6m) 3m ago 28m 67.2M - 2.43.0 a07b618ecd1d a5a043588d6d 2024-03-22T05:48:08.935 INFO:teuthology.orchestra.run.smithi005.stdout:rgw.foo.smithi005.exdhqu smithi005 *:8000 running (26m) 34s ago 26m 81.0M - 17.2.0 e1d6a67b021e 6bb27fe11553 2024-03-22T05:48:08.935 INFO:teuthology.orchestra.run.smithi005.stdout:rgw.foo.smithi186.mcgodk smithi186 *:8000 running (26m) 3m ago 26m 78.2M - 17.2.0 e1d6a67b021e 19b5c20a34f7 2024-03-22T05:48:09.352 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-03-22T05:48:09.352 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": { 2024-03-22T05:48:09.352 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev)": 3 2024-03-22T05:48:09.352 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:48:09.352 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr": { 2024-03-22T05:48:09.352 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev)": 2 2024-03-22T05:48:09.352 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:48:09.353 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": { 2024-03-22T05:48:09.353 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-03-22T05:48:09.353 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev)": 2 2024-03-22T05:48:09.353 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:48:09.353 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw": { 2024-03-22T05:48:09.353 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-22T05:48:09.353 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:48:09.353 INFO:teuthology.orchestra.run.smithi005.stdout: "overall": { 2024-03-22T05:48:09.353 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8, 2024-03-22T05:48:09.353 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev)": 7 2024-03-22T05:48:09.353 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-03-22T05:48:09.353 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-03-22T05:48:09.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:09 smithi005 ceph-mon[84705]: from='client.64294 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:48:09.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:09 smithi005 ceph-mon[89904]: from='client.64294 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:48:09.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:09 smithi186 ceph-mon[67903]: from='client.64294 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:48:09.698 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-03-22T05:48:09.698 INFO:teuthology.orchestra.run.smithi005.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37", 2024-03-22T05:48:09.698 INFO:teuthology.orchestra.run.smithi005.stdout: "in_progress": true, 2024-03-22T05:48:09.698 INFO:teuthology.orchestra.run.smithi005.stdout: "which": "Upgrading daemons of type(s) crash,osd. Upgrade limited to 1 daemons (1 remaining).", 2024-03-22T05:48:09.699 INFO:teuthology.orchestra.run.smithi005.stdout: "services_complete": [], 2024-03-22T05:48:09.699 INFO:teuthology.orchestra.run.smithi005.stdout: "progress": "", 2024-03-22T05:48:09.699 INFO:teuthology.orchestra.run.smithi005.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37 image", 2024-03-22T05:48:09.699 INFO:teuthology.orchestra.run.smithi005.stdout: "is_paused": false 2024-03-22T05:48:09.699 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-03-22T05:48:10.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:10 smithi005 ceph-mon[89904]: from='client.64300 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:48:10.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:10 smithi005 ceph-mon[89904]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:48:10.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:10 smithi005 ceph-mon[89904]: from='client.54372 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:48:10.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:10 smithi005 ceph-mon[89904]: from='client.? 172.21.15.5:0/733546810' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:48:10.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:10 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:48:10.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:10 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:48:10.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:10 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:48:10.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:10 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:48:10.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:10 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:48:10.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:10 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:48:10.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:10 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:48:10.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:10 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:48:10.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:10 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-22T05:48:10.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:10 smithi005 ceph-mon[84705]: from='client.64300 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:48:10.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:10 smithi005 ceph-mon[84705]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:48:10.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:10 smithi005 ceph-mon[84705]: from='client.54372 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:48:10.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:10 smithi005 ceph-mon[84705]: from='client.? 172.21.15.5:0/733546810' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:48:10.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:10 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:48:10.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:10 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:48:10.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:10 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:48:10.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:10 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:48:10.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:10 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:48:10.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:10 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:48:10.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:10 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:48:10.455 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:10 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:48:10.455 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:10 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-22T05:48:10.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:10 smithi186 ceph-mon[67903]: from='client.64300 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:48:10.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:10 smithi186 ceph-mon[67903]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:48:10.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:10 smithi186 ceph-mon[67903]: from='client.54372 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:48:10.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:10 smithi186 ceph-mon[67903]: from='client.? 172.21.15.5:0/733546810' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:48:10.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:10 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:48:10.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:10 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:48:10.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:10 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:48:10.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:10 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:48:10.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:10 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:48:10.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:10 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:48:10.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:10 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:48:10.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:10 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:48:10.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:10 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-22T05:48:11.433 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:11 smithi005 ceph-mon[89904]: from='client.54384 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:48:11.433 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:11 smithi005 ceph-mon[89904]: Upgrade: Target is version 19.0.0-2299-gea01217f (squid) 2024-03-22T05:48:11.433 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:11 smithi005 ceph-mon[89904]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, digests ['quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9'] 2024-03-22T05:48:11.433 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:11 smithi005 ceph-mon[89904]: Upgrade: Setting container_image for all mgr 2024-03-22T05:48:11.433 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:11 smithi005 ceph-mon[89904]: Upgrade: Setting container_image for all mon 2024-03-22T05:48:11.433 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:11 smithi005 ceph-mon[89904]: Upgrade: Setting container_image for all crash 2024-03-22T05:48:11.433 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:11 smithi005 ceph-mon[89904]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-22T05:48:11.433 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:11 smithi005 ceph-mon[89904]: Upgrade: osd.2 is safe to restart 2024-03-22T05:48:11.434 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:11 smithi005 ceph-mon[84705]: from='client.54384 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:48:11.434 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:11 smithi005 ceph-mon[84705]: Upgrade: Target is version 19.0.0-2299-gea01217f (squid) 2024-03-22T05:48:11.434 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:11 smithi005 ceph-mon[84705]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, digests ['quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9'] 2024-03-22T05:48:11.434 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:11 smithi005 ceph-mon[84705]: Upgrade: Setting container_image for all mgr 2024-03-22T05:48:11.434 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:11 smithi005 ceph-mon[84705]: Upgrade: Setting container_image for all mon 2024-03-22T05:48:11.434 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:11 smithi005 ceph-mon[84705]: Upgrade: Setting container_image for all crash 2024-03-22T05:48:11.434 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:11 smithi005 ceph-mon[84705]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-22T05:48:11.434 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:11 smithi005 ceph-mon[84705]: Upgrade: osd.2 is safe to restart 2024-03-22T05:48:11.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:11 smithi186 ceph-mon[67903]: from='client.54384 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:48:11.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:11 smithi186 ceph-mon[67903]: Upgrade: Target is version 19.0.0-2299-gea01217f (squid) 2024-03-22T05:48:11.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:11 smithi186 ceph-mon[67903]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, digests ['quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9'] 2024-03-22T05:48:11.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:11 smithi186 ceph-mon[67903]: Upgrade: Setting container_image for all mgr 2024-03-22T05:48:11.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:11 smithi186 ceph-mon[67903]: Upgrade: Setting container_image for all mon 2024-03-22T05:48:11.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:11 smithi186 ceph-mon[67903]: Upgrade: Setting container_image for all crash 2024-03-22T05:48:11.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:11 smithi186 ceph-mon[67903]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-22T05:48:11.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:11 smithi186 ceph-mon[67903]: Upgrade: osd.2 is safe to restart 2024-03-22T05:48:12.336 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:12 smithi005 ceph-mon[84705]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:48:12.337 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:12 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:48:12.337 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:12 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-22T05:48:12.337 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:12 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:48:12.337 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:12 smithi005 ceph-mon[89904]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:48:12.337 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:12 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:48:12.337 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:12 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-22T05:48:12.338 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:12 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:48:12.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:12 smithi186 ceph-mon[67903]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:48:12.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:12 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:48:12.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:12 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-22T05:48:12.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:12 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:48:13.338 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:13 smithi005 ceph-mon[84705]: Upgrade: Updating osd.2 2024-03-22T05:48:13.338 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:13 smithi005 ceph-mon[84705]: Deploying daemon osd.2 on smithi005 2024-03-22T05:48:13.338 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:13 smithi005 ceph-mon[89904]: Upgrade: Updating osd.2 2024-03-22T05:48:13.338 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:13 smithi005 ceph-mon[89904]: Deploying daemon osd.2 on smithi005 2024-03-22T05:48:13.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:13 smithi186 ceph-mon[67903]: Upgrade: Updating osd.2 2024-03-22T05:48:13.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:13 smithi186 ceph-mon[67903]: Deploying daemon osd.2 on smithi005 2024-03-22T05:48:14.366 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:14 smithi005 ceph-mon[89904]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 593 B/s rd, 0 op/s 2024-03-22T05:48:14.366 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:14 smithi005 ceph-mon[84705]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 593 B/s rd, 0 op/s 2024-03-22T05:48:14.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:14 smithi186 ceph-mon[67903]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 593 B/s rd, 0 op/s 2024-03-22T05:48:14.952 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 22 05:48:14 smithi005 systemd[1]: Stopping Ceph osd.2 for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:48:15.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:15 smithi005 ceph-mon[84705]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:48:15.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:15 smithi005 ceph-mon[84705]: osd.2 marked itself down and dead 2024-03-22T05:48:15.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:15 smithi005 ceph-mon[89904]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:48:15.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:15 smithi005 ceph-mon[89904]: osd.2 marked itself down and dead 2024-03-22T05:48:15.453 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 22 05:48:15 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-2[38242]: 2024-03-22T05:48:15.103+0000 7f859a7fe700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T05:48:15.453 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 22 05:48:15 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-2[38242]: 2024-03-22T05:48:15.103+0000 7f859a7fe700 -1 osd.2 115 *** Got signal Terminated *** 2024-03-22T05:48:15.453 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 22 05:48:15 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-2[38242]: 2024-03-22T05:48:15.103+0000 7f859a7fe700 -1 osd.2 115 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-22T05:48:15.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:15 smithi186 ceph-mon[67903]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:48:15.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:15 smithi186 ceph-mon[67903]: osd.2 marked itself down and dead 2024-03-22T05:48:15.929 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 22 05:48:15 smithi005 podman[109684]: 2024-03-22 05:48:15.644768016 +0000 UTC m=+0.669412739 container died 462ea63ad0e5a84c0f09583e715bc6cb2fc778228992329ea8ef31f5845ccfd1 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-2, distribution-scope=public, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, GIT_CLEAN=True, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_BRANCH=HEAD, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git) 2024-03-22T05:48:16.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:16 smithi005 ceph-mon[84705]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:48:16.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:16 smithi005 ceph-mon[84705]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-22T05:48:16.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:16 smithi005 ceph-mon[84705]: osdmap e116: 8 total, 7 up, 8 in 2024-03-22T05:48:16.203 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 22 05:48:16 smithi005 podman[109684]: 2024-03-22 05:48:16.120237436 +0000 UTC m=+1.144882211 container cleanup 462ea63ad0e5a84c0f09583e715bc6cb2fc778228992329ea8ef31f5845ccfd1 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-2, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, version=8, vcs-type=git, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, RELEASE=HEAD, com.redhat.component=centos-stream-container, distribution-scope=public, GIT_CLEAN=True, CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754) 2024-03-22T05:48:16.203 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 22 05:48:16 smithi005 bash[109684]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-2 2024-03-22T05:48:16.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:16 smithi005 ceph-mon[89904]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:48:16.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:16 smithi005 ceph-mon[89904]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-22T05:48:16.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:16 smithi005 ceph-mon[89904]: osdmap e116: 8 total, 7 up, 8 in 2024-03-22T05:48:16.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:16 smithi186 ceph-mon[67903]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:48:16.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:16 smithi186 ceph-mon[67903]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-22T05:48:16.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:16 smithi186 ceph-mon[67903]: osdmap e116: 8 total, 7 up, 8 in 2024-03-22T05:48:16.702 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 22 05:48:16 smithi005 podman[109703]: 2024-03-22 05:48:16.388616938 +0000 UTC m=+0.736148951 container remove 462ea63ad0e5a84c0f09583e715bc6cb2fc778228992329ea8ef31f5845ccfd1 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-2, build-date=2022-05-03T08:36:31.336870, release=754, RELEASE=HEAD, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, vcs-type=git, io.openshift.expose-services=, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, ceph=True, distribution-scope=public, com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux ) 2024-03-22T05:48:16.952 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 22 05:48:16 smithi005 podman[109796]: 2024-03-22 05:48:16.605398047 +0000 UTC m=+0.017876272 image pull aa461e7c5e7f322b169c789b389d594b994ea7ffcbc9e24e563a3ec9b69bc3c1 quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9 2024-03-22T05:48:16.952 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 22 05:48:16 smithi005 podman[109796]: 2024-03-22 05:48:16.714664638 +0000 UTC m=+0.127142889 container create 9626779e16885b60d555735e354401e6d0e76dd218a4ba412a4a81d40a009ed2 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-2-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, ceph=True, GIT_CLEAN=True) 2024-03-22T05:48:16.952 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 22 05:48:16 smithi005 podman[109796]: 2024-03-22 05:48:16.917317869 +0000 UTC m=+0.329796084 container init 9626779e16885b60d555735e354401e6d0e76dd218a4ba412a4a81d40a009ed2 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-2-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-22T05:48:16.952 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 22 05:48:16 smithi005 podman[109796]: 2024-03-22 05:48:16.920607252 +0000 UTC m=+0.333085466 container start 9626779e16885b60d555735e354401e6d0e76dd218a4ba412a4a81d40a009ed2 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-2-deactivate, org.label-schema.build-date=20240319, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-22T05:48:17.213 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:17 smithi005 ceph-mon[84705]: osdmap e117: 8 total, 7 up, 8 in 2024-03-22T05:48:17.213 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:16 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:48:16] "GET /metrics HTTP/1.1" 200 38231 "" "Prometheus/2.43.0" 2024-03-22T05:48:17.214 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 22 05:48:17 smithi005 podman[109796]: 2024-03-22 05:48:17.042114439 +0000 UTC m=+0.454592661 container attach 9626779e16885b60d555735e354401e6d0e76dd218a4ba412a4a81d40a009ed2 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-2-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-22T05:48:17.214 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:17 smithi005 ceph-mon[89904]: osdmap e117: 8 total, 7 up, 8 in 2024-03-22T05:48:17.495 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 22 05:48:17 smithi005 podman[109796]: 2024-03-22 05:48:17.223583565 +0000 UTC m=+0.636061771 container died 9626779e16885b60d555735e354401e6d0e76dd218a4ba412a4a81d40a009ed2 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-2-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=) 2024-03-22T05:48:17.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:17 smithi186 ceph-mon[67903]: osdmap e117: 8 total, 7 up, 8 in 2024-03-22T05:48:18.160 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 22 05:48:17 smithi005 podman[109831]: 2024-03-22 05:48:17.754522009 +0000 UTC m=+0.526075961 container remove 9626779e16885b60d555735e354401e6d0e76dd218a4ba412a4a81d40a009ed2 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-2-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0) 2024-03-22T05:48:18.160 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 22 05:48:17 smithi005 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@osd.2.service: Deactivated successfully. 2024-03-22T05:48:18.160 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 22 05:48:17 smithi005 systemd[1]: Stopped Ceph osd.2 for f59f36e2-e80a-11ee-95cd-87774f69a715. 2024-03-22T05:48:18.160 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 22 05:48:17 smithi005 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@osd.2.service: Consumed 12.451s CPU time. 2024-03-22T05:48:18.160 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 22 05:48:17 smithi005 systemd[1]: Starting Ceph osd.2 for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:48:18.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:18 smithi005 ceph-mon[84705]: pgmap v132: 161 pgs: 2 active+undersized+degraded, 10 stale+active+clean, 5 active+undersized, 144 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 7/723 objects degraded (0.968%) 2024-03-22T05:48:18.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:18 smithi005 ceph-mon[84705]: Health check failed: Degraded data redundancy: 7/723 objects degraded (0.968%), 2 pgs degraded (PG_DEGRADED) 2024-03-22T05:48:18.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:18 smithi005 ceph-mon[89904]: pgmap v132: 161 pgs: 2 active+undersized+degraded, 10 stale+active+clean, 5 active+undersized, 144 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 7/723 objects degraded (0.968%) 2024-03-22T05:48:18.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:18 smithi005 ceph-mon[89904]: Health check failed: Degraded data redundancy: 7/723 objects degraded (0.968%), 2 pgs degraded (PG_DEGRADED) 2024-03-22T05:48:18.453 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 22 05:48:18 smithi005 podman[109944]: 2024-03-22 05:48:18.091902503 +0000 UTC m=+0.017241943 image pull aa461e7c5e7f322b169c789b389d594b994ea7ffcbc9e24e563a3ec9b69bc3c1 quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9 2024-03-22T05:48:18.453 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 22 05:48:18 smithi005 podman[109944]: 2024-03-22 05:48:18.200119035 +0000 UTC m=+0.125458531 container create d55ab8d4a46160c419c731c37e10d9755725c9d8d50b2bae46ea6b96bda5fa55 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-2-activate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-03-22T05:48:18.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:18 smithi186 ceph-mon[67903]: pgmap v132: 161 pgs: 2 active+undersized+degraded, 10 stale+active+clean, 5 active+undersized, 144 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 7/723 objects degraded (0.968%) 2024-03-22T05:48:18.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:18 smithi186 ceph-mon[67903]: Health check failed: Degraded data redundancy: 7/723 objects degraded (0.968%), 2 pgs degraded (PG_DEGRADED) 2024-03-22T05:48:18.934 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 22 05:48:18 smithi005 podman[109944]: 2024-03-22 05:48:18.484452532 +0000 UTC m=+0.409791961 container init d55ab8d4a46160c419c731c37e10d9755725c9d8d50b2bae46ea6b96bda5fa55 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-2-activate, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-22T05:48:18.935 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 22 05:48:18 smithi005 podman[109944]: 2024-03-22 05:48:18.487579637 +0000 UTC m=+0.412919068 container start d55ab8d4a46160c419c731c37e10d9755725c9d8d50b2bae46ea6b96bda5fa55 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-2-activate, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-22T05:48:18.935 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 22 05:48:18 smithi005 podman[109944]: 2024-03-22 05:48:18.583917802 +0000 UTC m=+0.509257256 container attach d55ab8d4a46160c419c731c37e10d9755725c9d8d50b2bae46ea6b96bda5fa55 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-2-activate, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-22T05:48:19.203 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 22 05:48:18 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-2-activate[109960]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-22T05:48:19.203 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 22 05:48:18 smithi005 bash[109944]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-22T05:48:19.203 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 22 05:48:18 smithi005 bash[109944]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-22T05:48:19.203 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 22 05:48:18 smithi005 bash[109944]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-22T05:48:19.203 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 22 05:48:18 smithi005 bash[109944]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-22T05:48:19.203 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 22 05:48:18 smithi005 bash[109944]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-03-22T05:48:19.203 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 22 05:48:18 smithi005 bash[109944]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-22T05:48:19.203 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 22 05:48:18 smithi005 bash[109944]: --> ceph-volume raw activate successful for osd ID: 2 2024-03-22T05:48:19.203 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 22 05:48:18 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-2-activate[109960]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-22T05:48:19.204 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 22 05:48:18 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-2-activate[109960]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-22T05:48:19.204 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 22 05:48:18 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-2-activate[109960]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-22T05:48:19.204 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 22 05:48:18 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-2-activate[109960]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-03-22T05:48:19.204 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 22 05:48:18 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-2-activate[109960]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-22T05:48:19.204 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 22 05:48:18 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-2-activate[109960]: --> ceph-volume raw activate successful for osd ID: 2 2024-03-22T05:48:19.204 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 22 05:48:18 smithi005 podman[109944]: 2024-03-22 05:48:18.959960904 +0000 UTC m=+0.885300331 container died d55ab8d4a46160c419c731c37e10d9755725c9d8d50b2bae46ea6b96bda5fa55 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-2-activate, org.label-schema.build-date=20240319, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-22T05:48:19.934 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 22 05:48:19 smithi005 podman[110084]: 2024-03-22 05:48:19.596590008 +0000 UTC m=+0.627913699 container remove d55ab8d4a46160c419c731c37e10d9755725c9d8d50b2bae46ea6b96bda5fa55 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-2-activate, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-22T05:48:19.935 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 22 05:48:19 smithi005 podman[110149]: 2024-03-22 05:48:19.71132091 +0000 UTC m=+0.018636572 image pull aa461e7c5e7f322b169c789b389d594b994ea7ffcbc9e24e563a3ec9b69bc3c1 quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9 2024-03-22T05:48:19.935 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 22 05:48:19 smithi005 podman[110149]: 2024-03-22 05:48:19.814134301 +0000 UTC m=+0.121449960 container create 628204cb7c98642d56569171045efcb658293c1daaaf2448e9d6e12f142d9dda (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-22T05:48:20.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:20 smithi005 ceph-mon[84705]: pgmap v133: 161 pgs: 10 active+undersized+degraded, 2 stale+active+clean, 29 active+undersized, 120 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 41/723 objects degraded (5.671%) 2024-03-22T05:48:20.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:20 smithi005 ceph-mon[89904]: pgmap v133: 161 pgs: 10 active+undersized+degraded, 2 stale+active+clean, 29 active+undersized, 120 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 41/723 objects degraded (5.671%) 2024-03-22T05:48:20.453 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 22 05:48:20 smithi005 podman[110149]: 2024-03-22 05:48:20.066813672 +0000 UTC m=+0.374129332 container init 628204cb7c98642d56569171045efcb658293c1daaaf2448e9d6e12f142d9dda (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2) 2024-03-22T05:48:20.453 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 22 05:48:20 smithi005 podman[110149]: 2024-03-22 05:48:20.069883461 +0000 UTC m=+0.377199122 container start 628204cb7c98642d56569171045efcb658293c1daaaf2448e9d6e12f142d9dda (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-22T05:48:20.453 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 22 05:48:20 smithi005 ceph-osd[110168]: -- 172.21.15.5:0/4238060943 <== mon.0 v2:172.21.15.5:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56087ab4b680 con 0x56087ab15400 2024-03-22T05:48:20.453 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 22 05:48:20 smithi005 bash[110149]: 628204cb7c98642d56569171045efcb658293c1daaaf2448e9d6e12f142d9dda 2024-03-22T05:48:20.453 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 22 05:48:20 smithi005 systemd[1]: Started Ceph osd.2 for f59f36e2-e80a-11ee-95cd-87774f69a715. 2024-03-22T05:48:20.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:20 smithi186 ceph-mon[67903]: pgmap v133: 161 pgs: 10 active+undersized+degraded, 2 stale+active+clean, 29 active+undersized, 120 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 41/723 objects degraded (5.671%) 2024-03-22T05:48:20.952 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 22 05:48:20 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-2[110164]: 2024-03-22T05:48:20.613+0000 7f064667e7c0 -1 Falling back to public interface 2024-03-22T05:48:21.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:21 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:48:21.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:21 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:48:21.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:21 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:48:21.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:21 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:48:21.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:21 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:48:21.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:21 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:48:21.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:21 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:48:21.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:21 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:48:21.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:21 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:48:21.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:21 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:48:21.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:21 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:48:21.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:21 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:48:22.247 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:22 smithi005 ceph-mon[89904]: pgmap v134: 161 pgs: 12 active+undersized+degraded, 34 active+undersized, 115 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 63/723 objects degraded (8.714%) 2024-03-22T05:48:22.247 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:22 smithi005 ceph-mon[89904]: Health check update: Degraded data redundancy: 63/723 objects degraded (8.714%), 12 pgs degraded (PG_DEGRADED) 2024-03-22T05:48:22.248 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:22 smithi005 ceph-mon[84705]: pgmap v134: 161 pgs: 12 active+undersized+degraded, 34 active+undersized, 115 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 63/723 objects degraded (8.714%) 2024-03-22T05:48:22.248 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:22 smithi005 ceph-mon[84705]: Health check update: Degraded data redundancy: 63/723 objects degraded (8.714%), 12 pgs degraded (PG_DEGRADED) 2024-03-22T05:48:22.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:22 smithi186 ceph-mon[67903]: pgmap v134: 161 pgs: 12 active+undersized+degraded, 34 active+undersized, 115 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 63/723 objects degraded (8.714%) 2024-03-22T05:48:22.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:22 smithi186 ceph-mon[67903]: Health check update: Degraded data redundancy: 63/723 objects degraded (8.714%), 12 pgs degraded (PG_DEGRADED) 2024-03-22T05:48:23.581 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:23 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:48:23.581 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:23 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:48:23.581 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:23 smithi005 ceph-mon[84705]: pgmap v135: 161 pgs: 12 active+undersized+degraded, 34 active+undersized, 115 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 63/723 objects degraded (8.714%) 2024-03-22T05:48:23.581 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:23 smithi005 ceph-mon[84705]: from='osd.2 [v2:172.21.15.5:6818/2281158000,v1:172.21.15.5:6819/2281158000]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-22T05:48:23.581 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:23 smithi005 ceph-mon[84705]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-22T05:48:23.582 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:23 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:48:23.582 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:23 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:48:23.582 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:23 smithi005 ceph-mon[89904]: pgmap v135: 161 pgs: 12 active+undersized+degraded, 34 active+undersized, 115 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 63/723 objects degraded (8.714%) 2024-03-22T05:48:23.582 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:23 smithi005 ceph-mon[89904]: from='osd.2 [v2:172.21.15.5:6818/2281158000,v1:172.21.15.5:6819/2281158000]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-22T05:48:23.582 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:23 smithi005 ceph-mon[89904]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-22T05:48:23.582 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 22 05:48:23 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-2[110164]: 2024-03-22T05:48:23.325+0000 7f064667e7c0 -1 osd.2 0 read_superblock omap replica is missing. 2024-03-22T05:48:23.582 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 22 05:48:23 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-2[110164]: 2024-03-22T05:48:23.346+0000 7f064667e7c0 -1 osd.2 115 log_to_monitors true 2024-03-22T05:48:23.582 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 22 05:48:23 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-2[110164]: 2024-03-22T05:48:23.532+0000 7f0637ad6700 -1 osd.2 115 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-22T05:48:23.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:23 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:48:23.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:23 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:48:23.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:23 smithi186 ceph-mon[67903]: pgmap v135: 161 pgs: 12 active+undersized+degraded, 34 active+undersized, 115 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 63/723 objects degraded (8.714%) 2024-03-22T05:48:23.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:23 smithi186 ceph-mon[67903]: from='osd.2 [v2:172.21.15.5:6818/2281158000,v1:172.21.15.5:6819/2281158000]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-22T05:48:23.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:23 smithi186 ceph-mon[67903]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-22T05:48:24.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:24 smithi186 ceph-mon[67903]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-22T05:48:24.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:24 smithi186 ceph-mon[67903]: osdmap e118: 8 total, 7 up, 8 in 2024-03-22T05:48:24.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:24 smithi186 ceph-mon[67903]: from='osd.2 [v2:172.21.15.5:6818/2281158000,v1:172.21.15.5:6819/2281158000]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-03-22T05:48:24.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:24 smithi186 ceph-mon[67903]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-03-22T05:48:24.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:24 smithi186 ceph-mon[67903]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:48:24.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:24 smithi005 ceph-mon[89904]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-22T05:48:24.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:24 smithi005 ceph-mon[89904]: osdmap e118: 8 total, 7 up, 8 in 2024-03-22T05:48:24.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:24 smithi005 ceph-mon[89904]: from='osd.2 [v2:172.21.15.5:6818/2281158000,v1:172.21.15.5:6819/2281158000]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-03-22T05:48:24.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:24 smithi005 ceph-mon[89904]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-03-22T05:48:24.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:24 smithi005 ceph-mon[89904]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:48:24.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:24 smithi005 ceph-mon[84705]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-22T05:48:24.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:24 smithi005 ceph-mon[84705]: osdmap e118: 8 total, 7 up, 8 in 2024-03-22T05:48:24.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:24 smithi005 ceph-mon[84705]: from='osd.2 [v2:172.21.15.5:6818/2281158000,v1:172.21.15.5:6819/2281158000]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-03-22T05:48:24.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:24 smithi005 ceph-mon[84705]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-03-22T05:48:24.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:24 smithi005 ceph-mon[84705]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:48:25.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:25 smithi186 ceph-mon[67903]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-22T05:48:25.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:25 smithi186 ceph-mon[67903]: osd.2 [v2:172.21.15.5:6818/2281158000,v1:172.21.15.5:6819/2281158000] boot 2024-03-22T05:48:25.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:25 smithi186 ceph-mon[67903]: osdmap e119: 8 total, 8 up, 8 in 2024-03-22T05:48:25.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:25 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T05:48:25.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:25 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:48:25.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:25 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:48:25.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:25 smithi186 ceph-mon[67903]: pgmap v138: 161 pgs: 12 active+undersized+degraded, 34 active+undersized, 115 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 63/723 objects degraded (8.714%) 2024-03-22T05:48:25.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:25 smithi005 ceph-mon[84705]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-22T05:48:25.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:25 smithi005 ceph-mon[84705]: osd.2 [v2:172.21.15.5:6818/2281158000,v1:172.21.15.5:6819/2281158000] boot 2024-03-22T05:48:25.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:25 smithi005 ceph-mon[84705]: osdmap e119: 8 total, 8 up, 8 in 2024-03-22T05:48:25.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:25 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T05:48:25.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:25 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:48:25.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:25 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:48:25.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:25 smithi005 ceph-mon[84705]: pgmap v138: 161 pgs: 12 active+undersized+degraded, 34 active+undersized, 115 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 63/723 objects degraded (8.714%) 2024-03-22T05:48:25.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:25 smithi005 ceph-mon[89904]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-22T05:48:25.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:25 smithi005 ceph-mon[89904]: osd.2 [v2:172.21.15.5:6818/2281158000,v1:172.21.15.5:6819/2281158000] boot 2024-03-22T05:48:25.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:25 smithi005 ceph-mon[89904]: osdmap e119: 8 total, 8 up, 8 in 2024-03-22T05:48:25.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:25 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-22T05:48:25.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:25 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:48:25.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:25 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:48:25.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:25 smithi005 ceph-mon[89904]: pgmap v138: 161 pgs: 12 active+undersized+degraded, 34 active+undersized, 115 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 63/723 objects degraded (8.714%) 2024-03-22T05:48:26.648 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:26 smithi005 ceph-mon[84705]: osdmap e120: 8 total, 8 up, 8 in 2024-03-22T05:48:26.649 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:26 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:48:26.649 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:26 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:48:26.649 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:26 smithi005 ceph-mon[89904]: osdmap e120: 8 total, 8 up, 8 in 2024-03-22T05:48:26.649 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:26 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:48:26.649 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:26 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:48:26.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:26 smithi186 ceph-mon[67903]: osdmap e120: 8 total, 8 up, 8 in 2024-03-22T05:48:26.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:26 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:48:26.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:26 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:48:27.417 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:26 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:48:26] "GET /metrics HTTP/1.1" 200 38236 "" "Prometheus/2.43.0" 2024-03-22T05:48:27.685 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:27 smithi005 ceph-mon[84705]: pgmap v140: 161 pgs: 10 active+undersized+degraded, 31 active+undersized, 120 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 56/723 objects degraded (7.746%) 2024-03-22T05:48:27.686 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:27 smithi005 ceph-mon[89904]: pgmap v140: 161 pgs: 10 active+undersized+degraded, 31 active+undersized, 120 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 56/723 objects degraded (7.746%) 2024-03-22T05:48:27.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:27 smithi186 ceph-mon[67903]: pgmap v140: 161 pgs: 10 active+undersized+degraded, 31 active+undersized, 120 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 56/723 objects degraded (7.746%) 2024-03-22T05:48:28.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:28 smithi005 ceph-mon[84705]: Health check update: Degraded data redundancy: 56/723 objects degraded (7.746%), 10 pgs degraded (PG_DEGRADED) 2024-03-22T05:48:28.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:28 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:48:28.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:28 smithi005 ceph-mon[89904]: Health check update: Degraded data redundancy: 56/723 objects degraded (7.746%), 10 pgs degraded (PG_DEGRADED) 2024-03-22T05:48:28.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:28 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:48:28.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:28 smithi186 ceph-mon[67903]: Health check update: Degraded data redundancy: 56/723 objects degraded (7.746%), 10 pgs degraded (PG_DEGRADED) 2024-03-22T05:48:28.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:28 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:48:29.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:29 smithi005 ceph-mon[89904]: pgmap v141: 161 pgs: 2 active+undersized+degraded, 3 active+undersized, 156 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 22/723 objects degraded (3.043%) 2024-03-22T05:48:29.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:29 smithi005 ceph-mon[84705]: pgmap v141: 161 pgs: 2 active+undersized+degraded, 3 active+undersized, 156 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 22/723 objects degraded (3.043%) 2024-03-22T05:48:29.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:29 smithi186 ceph-mon[67903]: pgmap v141: 161 pgs: 2 active+undersized+degraded, 3 active+undersized, 156 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 22/723 objects degraded (3.043%) 2024-03-22T05:48:30.636 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:30 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:48:30.637 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:30 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:48:30.637 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:30 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:48:30.637 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:30 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:48:30.637 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:30 smithi005 ceph-mon[84705]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 847 B/s rd, 0 op/s 2024-03-22T05:48:30.637 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:30 smithi005 ceph-mon[84705]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1014 B/s rd, 0 op/s 2024-03-22T05:48:30.637 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:30 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:48:30.637 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:30 smithi005 ceph-mon[84705]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:48:30.637 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:30 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:48:30.637 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:30 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:48:30.637 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:30 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:48:30.637 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:30 smithi005 ceph-mon[84705]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:48:30.638 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:30 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:48:30.638 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:30 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:48:30.638 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:30 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:48:30.638 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:30 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:48:30.638 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:30 smithi005 ceph-mon[89904]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 847 B/s rd, 0 op/s 2024-03-22T05:48:30.638 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:30 smithi005 ceph-mon[89904]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1014 B/s rd, 0 op/s 2024-03-22T05:48:30.638 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:30 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:48:30.638 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:30 smithi005 ceph-mon[89904]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:48:30.638 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:30 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:48:30.638 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:30 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:48:30.638 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:30 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:48:30.638 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:30 smithi005 ceph-mon[89904]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:48:31.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:30 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:48:31.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:30 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:48:31.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:30 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:48:31.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:30 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:48:31.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:30 smithi186 ceph-mon[67903]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 847 B/s rd, 0 op/s 2024-03-22T05:48:31.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:30 smithi186 ceph-mon[67903]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1014 B/s rd, 0 op/s 2024-03-22T05:48:31.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:30 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:48:31.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:30 smithi186 ceph-mon[67903]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:48:31.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:30 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:48:31.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:30 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:48:31.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:30 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:48:31.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:30 smithi186 ceph-mon[67903]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:48:31.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[84705]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 22/723 objects degraded (3.043%), 2 pgs degraded) 2024-03-22T05:48:31.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:48:31.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:48:31.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:48:31.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:48:31.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:48:31.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:48:31.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:48:31.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:48:31.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:48:31.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:48:31.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-22T05:48:31.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-22T05:48:31.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-22T05:48:31.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:48:31.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:48:31.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-22T05:48:31.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-22T05:48:31.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-22T05:48:31.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-22T05:48:31.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-22T05:48:31.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-22T05:48:31.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-22T05:48:31.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-22T05:48:31.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-22T05:48:31.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-22T05:48:31.705 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[89904]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 22/723 objects degraded (3.043%), 2 pgs degraded) 2024-03-22T05:48:31.705 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:48:31.705 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:48:31.705 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:48:31.705 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:48:31.705 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:48:31.705 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:48:31.705 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:48:31.705 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:48:31.705 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:48:31.705 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:48:31.705 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-22T05:48:31.706 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-22T05:48:31.706 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-22T05:48:31.706 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:48:31.706 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:48:31.706 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-22T05:48:31.706 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-22T05:48:31.706 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-22T05:48:31.706 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-22T05:48:31.706 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-22T05:48:31.706 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-22T05:48:31.706 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-22T05:48:31.706 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-22T05:48:31.706 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-22T05:48:31.706 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-22T05:48:31.707 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:48:31.466+0000 7f7f0ecc9700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:48:31.707 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:48:31.707 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:48:31.707 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:48:31.707 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:48:31.707 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:48:31.707 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:48:31.707 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:48:31.707 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:48:31.707 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:48:31.707 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:48:31.708 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:48:31.708 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:48:31.708 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:48:31.708 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:48:31.708 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:48:31.708 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:48:31.708 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:48:31.708 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:48:31.709 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-22T05:48:31.709 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-22T05:48:31.709 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:48:31.709 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:48:31.709 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-22T05:48:31.709 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-22T05:48:31.709 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-22T05:48:31.709 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-22T05:48:31.709 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-22T05:48:31.709 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-22T05:48:31.709 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-22T05:48:31.709 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-22T05:48:31.709 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-22T05:48:31.710 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:48:31.710 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:48:31.710 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:48:31.710 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:48:31.710 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:48:31.710 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:48:31.710 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:48:31.710 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:48:31.710 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:48:31.710 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:48:31.710 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:48:31.710 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:48:31.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-22T05:48:31.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-22T05:48:31.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-22T05:48:31.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:48:31.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:48:31.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:48:31.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:48:31.711 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-22T05:48:31.712 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-22T05:48:31.712 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:48:31.712 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:48:31.712 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-22T05:48:31.712 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-22T05:48:31.712 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-22T05:48:31.712 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-22T05:48:31.712 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-22T05:48:31.712 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-22T05:48:31.712 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-22T05:48:31.712 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-22T05:48:31.712 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-22T05:48:31.712 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:48:31.712 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:48:31.713 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:48:31.713 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:48:31.713 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:48:31.713 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:48:31.713 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:48:31.713 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:48:31.713 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:48:31.713 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:48:31.713 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:48:31.713 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:48:31.713 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-22T05:48:31.713 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-22T05:48:31.713 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-22T05:48:31.713 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:48:31.713 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:48:31.714 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:48:31.714 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:48:31.714 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:48:31.714 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:48:31.714 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:48:31.714 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:48:31.714 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:48:31.714 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:48:31.714 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:48:31.715 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:48:31.715 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:48:31.715 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:48:31.715 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return future.result(timeout) 2024-03-22T05:48:31.715 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:48:31.715 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.__get_result() 2024-03-22T05:48:31.715 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:48:31.715 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: raise self._exception 2024-03-22T05:48:31.715 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:48:31.715 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: config_blobs=daemon_spec.final_config, 2024-03-22T05:48:31.715 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:48:31.715 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:48:31.715 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:48:31.715 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:48:31.715 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:48:31.716 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:48:31.716 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:48:31.716 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:48:31.716 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:48:31.716 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:48:31.716 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:48:31.716 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:48:31.716 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:48:31.716 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:48:31.716 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:48:31.716 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:48:31.716 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:48:31.716 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:48:31.716 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:48:31.716 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:48:31.717 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:48:31.717 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:48:31.717 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:31 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:48:32.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:31 smithi186 ceph-mon[67903]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 22/723 objects degraded (3.043%), 2 pgs degraded) 2024-03-22T05:48:32.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:31 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:48:32.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:31 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:48:32.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:31 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:48:32.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:31 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:48:32.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:31 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:48:32.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:31 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:48:32.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:31 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:48:32.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:31 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:48:32.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:31 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:48:32.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:31 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:48:32.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:31 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-22T05:48:32.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:31 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-22T05:48:32.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:31 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-22T05:48:32.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:31 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:48:32.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:31 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:48:32.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:31 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-22T05:48:32.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:31 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-22T05:48:32.046 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:31 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-22T05:48:32.046 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:31 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-22T05:48:32.046 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:31 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-22T05:48:32.046 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:31 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-22T05:48:32.046 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:31 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-22T05:48:32.046 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:31 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-22T05:48:32.046 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:31 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-22T05:48:32.046 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:31 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-22T05:48:32.046 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:31 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-22T05:48:32.046 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:31 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-22T05:48:32.046 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:31 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:48:32.046 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:31 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:48:32.046 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:31 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-22T05:48:32.047 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:31 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-22T05:48:32.047 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:31 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-22T05:48:32.047 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:31 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-22T05:48:32.047 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:31 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-22T05:48:32.047 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:31 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-22T05:48:32.047 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:31 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-22T05:48:32.047 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:31 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-22T05:48:32.047 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:31 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-22T05:48:32.047 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:31 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:48:32.047 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:31 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:48:32.047 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:31 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:48:32.047 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:31 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:48:32.047 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:31 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:48:32.047 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:31 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:48:32.048 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:31 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:48:32.048 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:31 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:48:32.048 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:31 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:48:32.048 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:31 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:48:32.048 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:31 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:48:32.048 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:31 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:48:32.048 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:31 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-22T05:48:32.048 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:31 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-22T05:48:32.048 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:31 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-22T05:48:32.048 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:31 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:48:32.048 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:31 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:48:32.048 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:31 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:48:32.048 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:31 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:48:32.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:32 smithi005 ceph-mon[84705]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:48:32.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:48:32.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:48:32.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:48:32.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:48:32.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:48:32.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:48:32.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:48:32.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:48:32.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:48:32.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:48:32.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:48:32.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:48:32.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:48:32.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:48:32.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:48:32.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:48:32.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:48:32.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:48:32.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:48:32.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:48:32.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:48:32.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:48:32.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:48:32.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:48:32.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:48:32.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:48:32.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:48:32.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:48:32.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: return future.result(timeout) 2024-03-22T05:48:32.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:48:32.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.__get_result() 2024-03-22T05:48:32.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:48:32.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: raise self._exception 2024-03-22T05:48:32.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:48:32.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:48:32.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:48:32.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:48:32.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:48:32.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:48:32.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:48:32.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:48:32.956 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:48:32.956 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:48:32.956 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:48:32.956 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:48:32.956 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:48:32.956 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:48:32.956 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:48:32.956 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:48:32.956 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:48:32.956 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:48:32.956 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:48:32.956 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:48:32.957 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:32 smithi005 ceph-mon[89904]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:48:32.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:48:32.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:48:32.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:48:32.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:48:32.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:48:32.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:48:32.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:48:32.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:48:32.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:48:32.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:48:32.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:48:32.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:48:32.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:48:32.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:48:32.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:48:32.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:48:32.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:48:32.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:48:32.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:48:32.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:48:32.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:48:32.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:48:32.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:48:32.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:48:32.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:48:32.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:48:32.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:48:32.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:48:32.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: return future.result(timeout) 2024-03-22T05:48:32.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:48:32.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.__get_result() 2024-03-22T05:48:32.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:48:32.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: raise self._exception 2024-03-22T05:48:32.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:48:32.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:48:32.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:48:32.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:48:32.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:48:32.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:48:32.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:48:32.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:48:32.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:48:32.960 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:48:32.960 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:48:32.960 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:48:32.960 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:48:32.960 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:48:32.960 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:48:32.960 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:48:32.960 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:48:32.960 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:48:32.960 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:48:32.960 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:48:32.960 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:48:32.961 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:48:32.961 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:48:32.961 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:48:32.961 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:48:32.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:32 smithi005 ceph-mon[84705]: Upgrade: Setting container_image for all nvmeof 2024-03-22T05:48:32.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:32 smithi005 ceph-mon[84705]: Upgrade: Finalizing container_image settings 2024-03-22T05:48:32.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:32 smithi005 ceph-mon[84705]: Upgrade: Complete! 2024-03-22T05:48:32.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:32 smithi005 ceph-mon[84705]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 745 B/s rd, 0 op/s 2024-03-22T05:48:32.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:32 smithi005 ceph-mon[84705]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 850 B/s rd, 0 op/s 2024-03-22T05:48:32.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:32 smithi005 ceph-mon[84705]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:48:32.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:32 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:48:32.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:32 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:48:32.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:32 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:48:32.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:32 smithi005 ceph-mon[84705]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:48:32.962 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:48:32.962 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:48:32.962 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:48:32.962 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:48:32.962 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:48:32.962 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:32 smithi005 ceph-mon[89904]: Upgrade: Setting container_image for all nvmeof 2024-03-22T05:48:32.962 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:32 smithi005 ceph-mon[89904]: Upgrade: Finalizing container_image settings 2024-03-22T05:48:32.962 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:32 smithi005 ceph-mon[89904]: Upgrade: Complete! 2024-03-22T05:48:32.962 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:32 smithi005 ceph-mon[89904]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 745 B/s rd, 0 op/s 2024-03-22T05:48:32.962 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:32 smithi005 ceph-mon[89904]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 850 B/s rd, 0 op/s 2024-03-22T05:48:32.962 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:32 smithi005 ceph-mon[89904]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:48:32.963 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:32 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:48:32.963 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:32 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:48:32.963 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:32 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:48:32.963 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:32 smithi005 ceph-mon[89904]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:48:33.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:32 smithi186 ceph-mon[67903]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:48:33.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:48:33.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:48:33.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:48:33.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:48:33.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:48:33.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:48:33.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:48:33.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:48:33.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:48:33.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:48:33.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:48:33.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:48:33.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:48:33.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:48:33.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:48:33.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:48:33.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:48:33.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:48:33.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:48:33.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:48:33.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:48:33.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:48:33.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:48:33.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:48:33.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:48:33.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:48:33.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:48:33.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:48:33.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: return future.result(timeout) 2024-03-22T05:48:33.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:48:33.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.__get_result() 2024-03-22T05:48:33.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:48:33.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: raise self._exception 2024-03-22T05:48:33.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:48:33.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:48:33.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:48:33.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:48:33.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:48:33.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:48:33.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:48:33.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:48:33.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:48:33.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:48:33.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:48:33.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:48:33.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:48:33.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:48:33.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:48:33.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:48:33.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:48:33.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:48:33.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:48:33.048 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:48:33.048 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:48:33.048 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:48:33.048 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:48:33.048 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:48:33.048 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:48:33.048 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:32 smithi186 ceph-mon[67903]: Upgrade: Setting container_image for all nvmeof 2024-03-22T05:48:33.048 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:32 smithi186 ceph-mon[67903]: Upgrade: Finalizing container_image settings 2024-03-22T05:48:33.048 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:32 smithi186 ceph-mon[67903]: Upgrade: Complete! 2024-03-22T05:48:33.048 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:32 smithi186 ceph-mon[67903]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 745 B/s rd, 0 op/s 2024-03-22T05:48:33.048 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:32 smithi186 ceph-mon[67903]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 850 B/s rd, 0 op/s 2024-03-22T05:48:33.048 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:32 smithi186 ceph-mon[67903]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:48:33.048 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:32 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:48:33.048 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:32 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:48:33.049 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:32 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:48:33.049 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:32 smithi186 ceph-mon[67903]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:48:33.590 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:33 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:48:33.496+0000 7f7f0ecc9700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:48:33.590 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:33 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:48:33.590 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:33 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:48:33.590 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:33 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:48:33.590 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:33 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:48:33.590 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:33 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:48:33.590 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:33 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:48:33.591 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:33 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:48:33.591 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:33 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:48:33.591 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:33 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:48:33.591 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:33 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:48:33.591 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:33 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:48:33.591 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:33 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:48:33.591 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:33 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:48:33.591 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:33 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:48:33.591 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:33 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:48:33.591 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:33 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:48:33.591 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:33 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:48:33.591 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:33 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:48:33.592 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:33 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:48:33.592 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:33 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:48:33.592 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:33 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:48:33.592 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:33 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:48:33.592 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:33 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:48:33.592 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:33 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:48:33.593 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:33 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:48:33.593 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:33 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:48:33.593 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:33 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:48:33.593 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:33 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:48:33.593 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:33 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return future.result(timeout) 2024-03-22T05:48:33.593 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:33 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:48:33.593 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:33 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.__get_result() 2024-03-22T05:48:33.593 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:33 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:48:33.593 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:33 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: raise self._exception 2024-03-22T05:48:33.593 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:33 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:48:33.593 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:33 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: config_blobs=daemon_spec.final_config, 2024-03-22T05:48:33.593 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:33 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:48:33.593 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:33 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:48:33.593 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:33 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:48:33.593 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:33 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:48:33.594 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:33 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:48:33.594 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:33 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:48:33.594 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:33 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:48:33.594 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:33 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:48:33.594 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:33 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:48:33.594 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:33 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:48:33.594 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:33 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:48:33.594 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:33 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:48:33.594 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:33 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:48:33.594 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:33 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:48:33.594 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:33 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:48:33.594 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:33 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:48:33.594 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:33 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:48:33.594 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:33 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:48:33.595 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:33 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:48:33.595 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:33 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:48:33.595 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:33 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:48:33.595 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:33 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:48:33.595 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:33 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:48:33.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:33 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:48:33.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:33 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:48:33.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:33 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:48:33.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:33 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:48:33.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:33 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:48:33.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:33 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:48:33.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:33 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:48:33.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:33 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:48:33.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:33 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:48:33.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:33 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:48:34.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:33 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:48:34.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:33 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:48:34.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:33 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:48:34.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:33 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:48:34.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:33 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:48:34.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:34 smithi005 ceph-mon[84705]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:48:34.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:48:34.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:48:34.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:48:34.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:48:34.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:48:34.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:48:34.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:48:34.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:48:34.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:48:34.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:48:34.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:48:34.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:48:34.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:48:34.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:48:34.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:48:34.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:48:34.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:48:34.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:48:34.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:48:34.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:48:34.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:48:34.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:48:34.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:48:34.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:48:34.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:48:34.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:48:34.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:48:34.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:48:34.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: return future.result(timeout) 2024-03-22T05:48:34.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:48:34.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.__get_result() 2024-03-22T05:48:34.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:48:34.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: raise self._exception 2024-03-22T05:48:34.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:48:34.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:48:34.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:48:34.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:48:34.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:48:34.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:48:34.956 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:48:34.956 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:48:34.956 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:48:34.956 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:48:34.956 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:48:34.956 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:48:34.956 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:48:34.956 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:48:34.956 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:48:34.956 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:48:34.956 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:48:34.956 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:48:34.956 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:48:34.956 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:48:34.957 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:34 smithi005 ceph-mon[89904]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:48:34.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:48:34.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:48:34.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:48:34.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:48:34.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:48:34.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:48:34.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:48:34.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:48:34.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:48:34.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:48:34.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:48:34.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:48:34.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:48:34.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:48:34.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:48:34.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:48:34.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:48:34.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:48:34.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:48:34.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:48:34.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:48:34.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:48:34.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:48:34.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:48:34.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:48:34.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:48:34.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:48:34.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:48:34.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: return future.result(timeout) 2024-03-22T05:48:34.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:48:34.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.__get_result() 2024-03-22T05:48:34.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:48:34.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: raise self._exception 2024-03-22T05:48:34.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:48:34.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:48:34.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:48:34.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:48:34.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:48:34.960 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:48:34.960 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:48:34.960 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:48:34.960 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:48:34.960 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:48:34.960 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:48:34.960 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:48:34.960 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:48:34.960 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:48:34.960 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:48:34.960 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:48:34.960 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:48:34.960 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:48:34.961 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:48:34.961 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:48:34.961 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:48:34.961 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:48:34.961 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:48:34.961 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:48:34.961 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:48:34.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:34 smithi005 ceph-mon[84705]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-22T05:48:34.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:34 smithi005 ceph-mon[84705]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-22T05:48:34.961 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:34 smithi005 ceph-mon[84705]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:48:34.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:34 smithi005 ceph-mon[84705]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:48:34.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:34 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:48:34.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:34 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:48:34.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:34 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:48:34.962 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:34 smithi005 ceph-mon[84705]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:48:34.962 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:48:34.962 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:48:34.962 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:48:34.962 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:48:34.962 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:48:34.962 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:34 smithi005 ceph-mon[89904]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-22T05:48:34.963 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:34 smithi005 ceph-mon[89904]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-22T05:48:34.963 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:34 smithi005 ceph-mon[89904]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:48:34.963 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:34 smithi005 ceph-mon[89904]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:48:34.963 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:34 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:48:34.963 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:34 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:48:34.963 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:34 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:48:34.963 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:34 smithi005 ceph-mon[89904]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:48:35.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:34 smithi186 ceph-mon[67903]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:48:35.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:48:35.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:48:35.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:48:35.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:48:35.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:48:35.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:48:35.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:48:35.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:48:35.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:48:35.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:48:35.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:48:35.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:48:35.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:48:35.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:48:35.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:48:35.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:48:35.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:48:35.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:48:35.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:48:35.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:48:35.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:48:35.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:48:35.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:48:35.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:48:35.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:48:35.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:48:35.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:48:35.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:48:35.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: return future.result(timeout) 2024-03-22T05:48:35.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:48:35.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.__get_result() 2024-03-22T05:48:35.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:48:35.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: raise self._exception 2024-03-22T05:48:35.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:48:35.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:48:35.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:48:35.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:48:35.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:48:35.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:48:35.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:48:35.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:48:35.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:48:35.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:48:35.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:48:35.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:48:35.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:48:35.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:48:35.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:48:35.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:48:35.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:48:35.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:48:35.048 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:48:35.048 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:48:35.048 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:48:35.048 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:48:35.048 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:48:35.048 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:48:35.048 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:48:35.048 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:34 smithi186 ceph-mon[67903]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-22T05:48:35.048 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:34 smithi186 ceph-mon[67903]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-22T05:48:35.048 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:34 smithi186 ceph-mon[67903]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:48:35.048 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:34 smithi186 ceph-mon[67903]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:48:35.048 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:34 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:48:35.048 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:34 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:48:35.048 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:34 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:48:35.049 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:34 smithi186 ceph-mon[67903]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:48:35.703 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:48:35.464+0000 7f7f0ecc9700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:48:35.703 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:48:35.703 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:48:35.703 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:48:35.703 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:48:35.703 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:48:35.703 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:48:35.703 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:48:35.704 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:48:35.704 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:48:35.704 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:48:35.704 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:48:35.704 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:48:35.704 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:48:35.704 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:48:35.704 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:48:35.704 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:48:35.704 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:48:35.704 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:48:35.705 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:48:35.705 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:48:35.705 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:48:35.705 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:48:35.705 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:48:35.705 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:48:35.705 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:48:35.705 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:48:35.705 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:48:35.706 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:48:35.706 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return future.result(timeout) 2024-03-22T05:48:35.706 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:48:35.706 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.__get_result() 2024-03-22T05:48:35.706 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:48:35.706 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: raise self._exception 2024-03-22T05:48:35.706 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:48:35.706 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: config_blobs=daemon_spec.final_config, 2024-03-22T05:48:35.706 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:48:35.706 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:48:35.706 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:48:35.706 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:48:35.706 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:48:35.706 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:48:35.707 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:48:35.707 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:48:35.707 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:48:35.707 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:48:35.707 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:48:35.707 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:48:35.707 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:48:35.707 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:48:35.707 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:48:35.707 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:48:35.707 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:48:35.707 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:48:35.707 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:48:35.707 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:48:35.708 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:48:35.708 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:48:35.708 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:35 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:48:36.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:36 smithi005 ceph-mon[84705]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:48:36.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:48:36.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:48:36.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:48:36.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:48:36.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:48:36.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:48:36.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:48:36.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:48:36.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:48:36.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:48:36.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:48:36.953 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:48:36.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:48:36.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:48:36.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:48:36.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:48:36.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:48:36.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:48:36.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:48:36.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:48:36.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:48:36.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:48:36.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:48:36.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:48:36.954 INFO:journalctl@ceph.mon.a.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:48:36.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:48:36.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:48:36.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:48:36.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: return future.result(timeout) 2024-03-22T05:48:36.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:48:36.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.__get_result() 2024-03-22T05:48:36.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:48:36.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: raise self._exception 2024-03-22T05:48:36.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:48:36.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:48:36.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:48:36.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:48:36.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:48:36.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:48:36.955 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:48:36.956 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:48:36.956 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:48:36.956 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:48:36.956 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:48:36.956 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:48:36.956 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:48:36.956 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:48:36.956 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:48:36.956 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:48:36.956 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:48:36.956 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:48:36.956 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:48:36.956 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:48:36.956 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:48:36.957 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:48:36.957 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:48:36.957 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:48:36.957 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:48:36.957 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:36 smithi005 ceph-mon[84705]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 860 B/s rd, 0 op/s 2024-03-22T05:48:36.957 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:36 smithi005 ceph-mon[89904]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:48:36.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:48:36.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:48:36.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:48:36.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:48:36.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:48:36.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:48:36.957 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:48:36.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:48:36.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:48:36.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:48:36.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:48:36.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:48:36.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:48:36.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:48:36.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:48:36.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:48:36.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:48:36.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:48:36.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:48:36.958 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:48:36.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:48:36.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:48:36.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:48:36.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:48:36.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:48:36.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:48:36.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:48:36.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:48:36.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: return future.result(timeout) 2024-03-22T05:48:36.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:48:36.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.__get_result() 2024-03-22T05:48:36.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:48:36.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: raise self._exception 2024-03-22T05:48:36.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:48:36.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:48:36.959 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:48:36.960 INFO:journalctl@ceph.mon.c.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:48:36.960 INFO:journalctl@ceph.mon.c.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:48:36.960 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:48:36.960 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:48:36.960 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:48:36.960 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:48:36.960 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:48:36.960 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:48:36.960 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:48:36.960 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:48:36.960 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:48:36.960 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:48:36.960 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:48:36.960 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:48:36.961 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:48:36.961 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:48:36.961 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:48:36.961 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:48:36.961 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:48:36.961 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:48:36.961 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:48:36.961 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:48:36.961 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:36 smithi005 ceph-mon[89904]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 860 B/s rd, 0 op/s 2024-03-22T05:48:37.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:36 smithi186 ceph-mon[67903]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:48:37.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:48:37.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:48:37.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:48:37.044 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:48:37.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:48:37.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:48:37.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:48:37.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:48:37.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:48:37.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:48:37.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:48:37.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:48:37.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:48:37.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:48:37.045 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:48:37.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:48:37.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:48:37.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:48:37.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:48:37.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:48:37.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:48:37.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:48:37.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:48:37.046 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:48:37.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:48:37.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:48:37.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:48:37.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:48:37.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: return future.result(timeout) 2024-03-22T05:48:37.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:48:37.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.__get_result() 2024-03-22T05:48:37.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:48:37.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: raise self._exception 2024-03-22T05:48:37.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:48:37.047 INFO:journalctl@ceph.mon.b.smithi186.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:48:37.048 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:48:37.048 INFO:journalctl@ceph.mon.b.smithi186.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:48:37.048 INFO:journalctl@ceph.mon.b.smithi186.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:48:37.048 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:48:37.048 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:48:37.048 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:48:37.048 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:48:37.048 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:48:37.048 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:48:37.049 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:48:37.049 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:48:37.049 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:48:37.049 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:48:37.049 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:48:37.049 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:48:37.049 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:48:37.049 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:48:37.049 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:48:37.050 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:48:37.050 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:48:37.050 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:48:37.050 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:48:37.050 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:48:37.050 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:36 smithi186 ceph-mon[67903]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 860 B/s rd, 0 op/s 2024-03-22T05:48:37.452 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:36 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:48:36] "GET /metrics HTTP/1.1" 200 38236 "" "Prometheus/2.43.0" 2024-03-22T05:48:38.774 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:38 smithi005 ceph-mon[84705]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 644 B/s rd, 0 op/s 2024-03-22T05:48:38.774 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:38 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:48:38.774 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:38 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:48:38.774 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:38 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:48:38.774 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:38 smithi005 ceph-mon[89904]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 644 B/s rd, 0 op/s 2024-03-22T05:48:38.775 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:38 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:48:38.775 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:38 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:48:38.775 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:38 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:48:38.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:38 smithi186 ceph-mon[67903]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 644 B/s rd, 0 op/s 2024-03-22T05:48:38.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:38 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:48:38.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:38 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:48:38.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:38 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:48:40.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:40 smithi005 ceph-mon[84705]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-22T05:48:40.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:40 smithi005 ceph-mon[84705]: from='client.64330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:48:40.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:40 smithi005 ceph-mon[89904]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-22T05:48:40.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:40 smithi005 ceph-mon[89904]: from='client.64330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:48:40.778 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph orch ps' 2024-03-22T05:48:40.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:40 smithi186 ceph-mon[67903]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-22T05:48:40.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:40 smithi186 ceph-mon[67903]: from='client.64330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:48:42.734 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:42 smithi005 ceph-mon[84705]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-22T05:48:42.734 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:42 smithi005 ceph-mon[89904]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-22T05:48:42.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:42 smithi186 ceph-mon[67903]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-22T05:48:42.951 INFO:teuthology.orchestra.run.smithi005.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T05:48:42.951 INFO:teuthology.orchestra.run.smithi005.stdout:alertmanager.a smithi005 *:9093,9094 running (6m) 18s ago 28m 28.0M - 0.25.0 c8568f914cd2 ee99dfc435e8 2024-03-22T05:48:42.951 INFO:teuthology.orchestra.run.smithi005.stdout:grafana.a smithi186 *:3000 running (6m) 3m ago 27m 81.5M - 9.4.12 2bacad6d85d8 16fd57dbbd71 2024-03-22T05:48:42.951 INFO:teuthology.orchestra.run.smithi005.stdout:iscsi.foo.smithi005.mcdwbp smithi005 unknown 18s ago 27m 64.6M - 3.5 e1d6a67b021e 5215e03bf81b 2024-03-22T05:48:42.951 INFO:teuthology.orchestra.run.smithi005.stdout:mgr.x smithi186 *:8443,9283,8765 running (7m) 3m ago 32m 448M - 19.0.0-2299-gea01217f aa461e7c5e7f c556421a73f3 2024-03-22T05:48:42.951 INFO:teuthology.orchestra.run.smithi005.stdout:mgr.y smithi005 *:8443,9283,8765 running (17m) 18s ago 34m 509M - 19.0.0-2299-gea01217f aa461e7c5e7f 62a813eebc0d 2024-03-22T05:48:42.951 INFO:teuthology.orchestra.run.smithi005.stdout:mon.a smithi005 running (4m) 18s ago 34m 52.9M 2048M 19.0.0-2299-gea01217f aa461e7c5e7f 111a2aa24dd9 2024-03-22T05:48:42.952 INFO:teuthology.orchestra.run.smithi005.stdout:mon.b smithi186 running (4m) 3m ago 32m 39.1M 2048M 19.0.0-2299-gea01217f aa461e7c5e7f c1798feac15b 2024-03-22T05:48:42.952 INFO:teuthology.orchestra.run.smithi005.stdout:mon.c smithi005 running (3m) 18s ago 33m 40.7M 2048M 19.0.0-2299-gea01217f aa461e7c5e7f 708c72ff74f5 2024-03-22T05:48:42.952 INFO:teuthology.orchestra.run.smithi005.stdout:node-exporter.a smithi005 *:9100 running (6m) 18s ago 28m 16.5M - 1.5.0 0da6a335fe13 f635cd72b9d3 2024-03-22T05:48:42.952 INFO:teuthology.orchestra.run.smithi005.stdout:node-exporter.b smithi186 *:9100 running (6m) 3m ago 28m 15.1M - 1.5.0 0da6a335fe13 d7e0b3d20191 2024-03-22T05:48:42.952 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0 smithi005 running (110s) 18s ago 32m 43.2M 2993M 19.0.0-2299-gea01217f aa461e7c5e7f 31b6cdee5f5b 2024-03-22T05:48:42.952 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1 smithi005 running (73s) 18s ago 31m 43.7M 2993M 19.0.0-2299-gea01217f aa461e7c5e7f 7d21bc9fd3c2 2024-03-22T05:48:42.952 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2 smithi005 running (23s) 18s ago 31m 12.7M 2993M 19.0.0-2299-gea01217f aa461e7c5e7f 628204cb7c98 2024-03-22T05:48:42.952 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3 smithi005 running (31m) 18s ago 31m 58.4M 2993M 17.2.0 e1d6a67b021e b25432a7239b 2024-03-22T05:48:42.952 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4 smithi186 running (30m) 3m ago 30m 56.4M 3248M 17.2.0 e1d6a67b021e cd963671f340 2024-03-22T05:48:42.952 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5 smithi186 running (30m) 3m ago 30m 51.7M 3248M 17.2.0 e1d6a67b021e f98c87f1a2cd 2024-03-22T05:48:42.952 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6 smithi186 running (30m) 3m ago 30m 52.0M 3248M 17.2.0 e1d6a67b021e 704ac49c68c4 2024-03-22T05:48:42.952 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7 smithi186 running (29m) 3m ago 29m 54.3M 3248M 17.2.0 e1d6a67b021e f415b90edf63 2024-03-22T05:48:42.952 INFO:teuthology.orchestra.run.smithi005.stdout:prometheus.a smithi186 *:9095 running (6m) 3m ago 28m 67.2M - 2.43.0 a07b618ecd1d a5a043588d6d 2024-03-22T05:48:42.952 INFO:teuthology.orchestra.run.smithi005.stdout:rgw.foo.smithi005.exdhqu smithi005 *:8000 running (27m) 18s ago 27m 81.2M - 17.2.0 e1d6a67b021e 6bb27fe11553 2024-03-22T05:48:42.952 INFO:teuthology.orchestra.run.smithi005.stdout:rgw.foo.smithi186.mcgodk smithi186 *:8000 running (27m) 3m ago 27m 78.2M - 17.2.0 e1d6a67b021e 19b5c20a34f7 2024-03-22T05:48:43.646 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:43 smithi005 ceph-mon[84705]: from='client.54402 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:48:43.646 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:43 smithi005 ceph-mon[89904]: from='client.54402 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:48:43.703 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph versions | jq -e '"'"'.osd | length == 2'"'"'' 2024-03-22T05:48:43.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:43 smithi186 ceph-mon[67903]: from='client.54402 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:48:44.690 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:44 smithi005 ceph-mon[89904]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:48:44.690 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:44 smithi005 ceph-mon[89904]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:48:44.690 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:44 smithi005 ceph-mon[84705]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:48:44.690 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:44 smithi005 ceph-mon[84705]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:48:44.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:44 smithi186 ceph-mon[67903]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:48:44.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:44 smithi186 ceph-mon[67903]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:48:45.774 INFO:teuthology.orchestra.run.smithi005.stdout:true 2024-03-22T05:48:46.532 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '"'"'.up_to_date | length == 8'"'"'' 2024-03-22T05:48:46.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:46 smithi186 ceph-mon[67903]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:48:46.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:46 smithi186 ceph-mon[67903]: from='client.? 172.21.15.5:0/668734805' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:48:46.848 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:46 smithi005 ceph-mon[84705]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:48:46.848 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:46 smithi005 ceph-mon[84705]: from='client.? 172.21.15.5:0/668734805' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:48:46.848 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:46 smithi005 ceph-mon[89904]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:48:46.848 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:46 smithi005 ceph-mon[89904]: from='client.? 172.21.15.5:0/668734805' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:48:47.174 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:46 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:48:46] "GET /metrics HTTP/1.1" 200 38243 "" "Prometheus/2.43.0" 2024-03-22T05:48:48.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:48 smithi186 ceph-mon[67903]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:48:48.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:48 smithi005 ceph-mon[84705]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:48:48.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:48 smithi005 ceph-mon[89904]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:48:49.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:49 smithi186 ceph-mon[67903]: from='client.54414 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:48:49.854 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:49 smithi005 ceph-mon[84705]: from='client.54414 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:48:49.854 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:49 smithi005 ceph-mon[89904]: from='client.54414 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:48:50.694 INFO:teuthology.orchestra.run.smithi005.stdout:true 2024-03-22T05:48:50.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:50 smithi186 ceph-mon[67903]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:48:50.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:50 smithi005 ceph-mon[84705]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:48:50.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:50 smithi005 ceph-mon[89904]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:48:51.542 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph orch upgrade status' 2024-03-22T05:48:52.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:52 smithi186 ceph-mon[67903]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:48:52.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:52 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:48:52.819 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:52 smithi005 ceph-mon[84705]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:48:52.819 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:52 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:48:52.819 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:52 smithi005 ceph-mon[89904]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:48:52.819 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:52 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:48:53.583 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-03-22T05:48:53.583 INFO:teuthology.orchestra.run.smithi005.stdout: "target_image": null, 2024-03-22T05:48:53.584 INFO:teuthology.orchestra.run.smithi005.stdout: "in_progress": false, 2024-03-22T05:48:53.584 INFO:teuthology.orchestra.run.smithi005.stdout: "which": "", 2024-03-22T05:48:53.584 INFO:teuthology.orchestra.run.smithi005.stdout: "services_complete": [], 2024-03-22T05:48:53.584 INFO:teuthology.orchestra.run.smithi005.stdout: "progress": null, 2024-03-22T05:48:53.584 INFO:teuthology.orchestra.run.smithi005.stdout: "message": "", 2024-03-22T05:48:53.584 INFO:teuthology.orchestra.run.smithi005.stdout: "is_paused": false 2024-03-22T05:48:53.584 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-03-22T05:48:54.252 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph health detail' 2024-03-22T05:48:54.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:54 smithi186 ceph-mon[67903]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:48:54.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:54 smithi186 ceph-mon[67903]: from='client.64354 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:48:54.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:54 smithi186 ceph-mon[67903]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:48:54.885 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:54 smithi005 ceph-mon[84705]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:48:54.886 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:54 smithi005 ceph-mon[84705]: from='client.64354 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:48:54.886 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:54 smithi005 ceph-mon[84705]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:48:54.886 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:54 smithi005 ceph-mon[89904]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:48:54.886 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:54 smithi005 ceph-mon[89904]: from='client.64354 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:48:54.886 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:54 smithi005 ceph-mon[89904]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:48:56.288 INFO:teuthology.orchestra.run.smithi005.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-22T05:48:56.288 INFO:teuthology.orchestra.run.smithi005.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-22T05:48:56.288 INFO:teuthology.orchestra.run.smithi005.stdout: daemon iscsi.foo.smithi005.mcdwbp on smithi005 is in unknown state 2024-03-22T05:48:56.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:56 smithi186 ceph-mon[67903]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:48:56.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:56 smithi186 ceph-mon[67903]: from='client.? 172.21.15.5:0/3352570281' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T05:48:56.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:56 smithi005 ceph-mon[84705]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:48:56.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:56 smithi005 ceph-mon[84705]: from='client.? 172.21.15.5:0/3352570281' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T05:48:56.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:56 smithi005 ceph-mon[89904]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:48:56.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:56 smithi005 ceph-mon[89904]: from='client.? 172.21.15.5:0/3352570281' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T05:48:57.062 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types crash,osd' 2024-03-22T05:48:57.376 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:48:56 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:48:56] "GET /metrics HTTP/1.1" 200 38245 "" "Prometheus/2.43.0" 2024-03-22T05:48:58.638 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:58 smithi005 ceph-mon[84705]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:48:58.638 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:58 smithi005 ceph-mon[89904]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:48:58.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:58 smithi186 ceph-mon[67903]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:48:59.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:48:59 smithi005 ceph-mon[84705]: from='client.64366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37", "daemon_types": "crash,osd", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:48:59.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:48:59 smithi005 ceph-mon[89904]: from='client.64366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37", "daemon_types": "crash,osd", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:49:00.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:48:59 smithi186 ceph-mon[67903]: from='client.64366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37", "daemon_types": "crash,osd", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:49:00.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:00 smithi005 ceph-mon[89904]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:49:00.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:00 smithi005 ceph-mon[84705]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:49:00.945 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:00 smithi186 ceph-mon[67903]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:49:01.333 INFO:teuthology.orchestra.run.smithi005.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37 2024-03-22T05:49:02.137 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-03-22T05:49:02.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:02 smithi005 ceph-mon[84705]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37 2024-03-22T05:49:02.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:02 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:49:02.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:02 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:49:02.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:02 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:49:02.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:02 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:49:02.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:02 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:49:02.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:02 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:49:02.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:02 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:49:02.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:02 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:49:02.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:02 smithi005 ceph-mon[89904]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37 2024-03-22T05:49:02.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:02 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:49:02.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:02 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:49:02.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:02 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:49:02.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:02 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:49:02.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:02 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:49:02.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:02 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:49:02.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:02 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:49:02.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:02 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:49:02.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:02 smithi186 ceph-mon[67903]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37 2024-03-22T05:49:02.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:02 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:49:02.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:02 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:49:02.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:02 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:49:02.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:02 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:49:02.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:02 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:49:02.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:02 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:49:02.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:02 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:49:02.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:02 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:49:03.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:03 smithi005 ceph-mon[84705]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:49:03.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:03 smithi005 ceph-mon[84705]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 984 B/s rd, 0 op/s 2024-03-22T05:49:03.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:03 smithi005 ceph-mon[84705]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 609 B/s rd, 0 op/s 2024-03-22T05:49:03.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:03 smithi005 ceph-mon[84705]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:49:03.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:03 smithi005 ceph-mon[84705]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:49:03.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:03 smithi005 ceph-mon[89904]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:49:03.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:03 smithi005 ceph-mon[89904]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 984 B/s rd, 0 op/s 2024-03-22T05:49:03.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:03 smithi005 ceph-mon[89904]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 609 B/s rd, 0 op/s 2024-03-22T05:49:03.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:03 smithi005 ceph-mon[89904]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:49:03.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:03 smithi005 ceph-mon[89904]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:49:03.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:03 smithi186 ceph-mon[67903]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:49:03.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:03 smithi186 ceph-mon[67903]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 984 B/s rd, 0 op/s 2024-03-22T05:49:03.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:03 smithi186 ceph-mon[67903]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 609 B/s rd, 0 op/s 2024-03-22T05:49:03.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:03 smithi186 ceph-mon[67903]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:49:03.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:03 smithi186 ceph-mon[67903]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:49:04.667 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:04 smithi005 ceph-mon[84705]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:49:04.668 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:04 smithi005 ceph-mon[84705]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:49:04.668 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:04 smithi005 ceph-mon[89904]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:49:04.668 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:04 smithi005 ceph-mon[89904]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:49:04.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:04 smithi186 ceph-mon[67903]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:49:04.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:04 smithi186 ceph-mon[67903]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:49:04.945 INFO:teuthology.orchestra.run.smithi005.stdout:true 2024-03-22T05:49:05.203 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:05 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:49:05.125+0000 7f7f0ecc9700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:49:05.203 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:05 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:49:05.204 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:05 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:49:05.204 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:05 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:49:05.204 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:05 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:49:05.204 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:05 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:49:05.204 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:05 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:49:05.204 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:05 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:49:05.204 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:05 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:49:05.204 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:05 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:49:05.205 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:05 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:49:05.205 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:05 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:49:05.205 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:05 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:49:05.205 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:05 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:49:05.205 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:05 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:49:05.205 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:05 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:49:05.205 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:05 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:49:05.205 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:05 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:49:05.205 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:05 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:49:05.206 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:05 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:49:05.206 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:05 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:49:05.207 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:05 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:49:05.207 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:05 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:49:05.207 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:05 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:49:05.207 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:05 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:49:05.207 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:05 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:49:05.207 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:05 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:49:05.207 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:05 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:49:05.207 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:05 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:49:05.208 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:05 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return future.result(timeout) 2024-03-22T05:49:05.208 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:05 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:49:05.208 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:05 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.__get_result() 2024-03-22T05:49:05.208 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:05 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:49:05.208 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:05 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: raise self._exception 2024-03-22T05:49:05.208 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:05 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:49:05.208 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:05 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: config_blobs=daemon_spec.final_config, 2024-03-22T05:49:05.208 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:05 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:49:05.208 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:05 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:49:05.209 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:05 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:49:05.209 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:05 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:49:05.209 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:05 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:49:05.209 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:05 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:49:05.209 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:05 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:49:05.209 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:05 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:49:05.209 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:05 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:49:05.209 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:05 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:49:05.209 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:05 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:49:05.210 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:05 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:49:05.210 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:05 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:49:05.210 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:05 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:49:05.210 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:05 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:49:05.210 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:05 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:49:05.210 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:05 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:49:05.210 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:05 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:49:05.210 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:05 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:49:05.210 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:05 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:49:05.211 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:05 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:49:05.211 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:05 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:49:05.211 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:05 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:49:05.638 INFO:teuthology.orchestra.run.smithi005.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T05:49:05.639 INFO:teuthology.orchestra.run.smithi005.stdout:alertmanager.a smithi005 *:9093,9094 running (6m) 41s ago 28m 28.0M - 0.25.0 c8568f914cd2 ee99dfc435e8 2024-03-22T05:49:05.639 INFO:teuthology.orchestra.run.smithi005.stdout:grafana.a smithi186 *:3000 running (6m) 4m ago 28m 81.5M - 9.4.12 2bacad6d85d8 16fd57dbbd71 2024-03-22T05:49:05.639 INFO:teuthology.orchestra.run.smithi005.stdout:iscsi.foo.smithi005.mcdwbp smithi005 unknown 41s ago 27m 64.6M - 3.5 e1d6a67b021e 5215e03bf81b 2024-03-22T05:49:05.639 INFO:teuthology.orchestra.run.smithi005.stdout:mgr.x smithi186 *:8443,9283,8765 running (7m) 4m ago 32m 448M - 19.0.0-2299-gea01217f aa461e7c5e7f c556421a73f3 2024-03-22T05:49:05.639 INFO:teuthology.orchestra.run.smithi005.stdout:mgr.y smithi005 *:8443,9283,8765 running (17m) 41s ago 34m 509M - 19.0.0-2299-gea01217f aa461e7c5e7f 62a813eebc0d 2024-03-22T05:49:05.639 INFO:teuthology.orchestra.run.smithi005.stdout:mon.a smithi005 running (4m) 41s ago 34m 52.9M 2048M 19.0.0-2299-gea01217f aa461e7c5e7f 111a2aa24dd9 2024-03-22T05:49:05.639 INFO:teuthology.orchestra.run.smithi005.stdout:mon.b smithi186 running (5m) 4m ago 33m 39.1M 2048M 19.0.0-2299-gea01217f aa461e7c5e7f c1798feac15b 2024-03-22T05:49:05.639 INFO:teuthology.orchestra.run.smithi005.stdout:mon.c smithi005 running (3m) 41s ago 33m 40.7M 2048M 19.0.0-2299-gea01217f aa461e7c5e7f 708c72ff74f5 2024-03-22T05:49:05.639 INFO:teuthology.orchestra.run.smithi005.stdout:node-exporter.a smithi005 *:9100 running (7m) 41s ago 29m 16.5M - 1.5.0 0da6a335fe13 f635cd72b9d3 2024-03-22T05:49:05.639 INFO:teuthology.orchestra.run.smithi005.stdout:node-exporter.b smithi186 *:9100 running (7m) 4m ago 29m 15.1M - 1.5.0 0da6a335fe13 d7e0b3d20191 2024-03-22T05:49:05.639 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0 smithi005 running (2m) 41s ago 32m 43.2M 2993M 19.0.0-2299-gea01217f aa461e7c5e7f 31b6cdee5f5b 2024-03-22T05:49:05.639 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1 smithi005 running (95s) 41s ago 32m 43.7M 2993M 19.0.0-2299-gea01217f aa461e7c5e7f 7d21bc9fd3c2 2024-03-22T05:49:05.639 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2 smithi005 running (45s) 41s ago 31m 12.7M 2993M 19.0.0-2299-gea01217f aa461e7c5e7f 628204cb7c98 2024-03-22T05:49:05.639 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3 smithi005 running (31m) 41s ago 31m 58.4M 2993M 17.2.0 e1d6a67b021e b25432a7239b 2024-03-22T05:49:05.639 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4 smithi186 running (31m) 4m ago 31m 56.4M 3248M 17.2.0 e1d6a67b021e cd963671f340 2024-03-22T05:49:05.640 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5 smithi186 running (30m) 4m ago 30m 51.7M 3248M 17.2.0 e1d6a67b021e f98c87f1a2cd 2024-03-22T05:49:05.640 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6 smithi186 running (30m) 4m ago 30m 52.0M 3248M 17.2.0 e1d6a67b021e 704ac49c68c4 2024-03-22T05:49:05.640 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7 smithi186 running (30m) 4m ago 29m 54.3M 3248M 17.2.0 e1d6a67b021e f415b90edf63 2024-03-22T05:49:05.640 INFO:teuthology.orchestra.run.smithi005.stdout:prometheus.a smithi186 *:9095 running (6m) 4m ago 29m 67.2M - 2.43.0 a07b618ecd1d a5a043588d6d 2024-03-22T05:49:05.640 INFO:teuthology.orchestra.run.smithi005.stdout:rgw.foo.smithi005.exdhqu smithi005 *:8000 running (27m) 41s ago 27m 81.2M - 17.2.0 e1d6a67b021e 6bb27fe11553 2024-03-22T05:49:05.640 INFO:teuthology.orchestra.run.smithi005.stdout:rgw.foo.smithi186.mcgodk smithi186 *:8000 running (27m) 4m ago 27m 78.2M - 17.2.0 e1d6a67b021e 19b5c20a34f7 2024-03-22T05:49:05.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:05 smithi005 ceph-mon[84705]: from='client.64372 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:49:05.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:05 smithi005 ceph-mon[84705]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:49:05.702 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:49:05.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:49:05.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:49:05.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:49:05.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:49:05.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:49:05.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:49:05.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:49:05.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:49:05.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:49:05.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:49:05.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:49:05.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:49:05.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:49:05.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:49:05.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:49:05.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:49:05.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:49:05.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:49:05.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:49:05.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:49:05.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:49:05.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:49:05.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:49:05.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:49:05.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:49:05.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:49:05.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:49:05.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: return future.result(timeout) 2024-03-22T05:49:05.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:49:05.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.__get_result() 2024-03-22T05:49:05.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:49:05.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: raise self._exception 2024-03-22T05:49:05.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:49:05.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:49:05.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:49:05.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:49:05.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:49:05.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:49:05.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:49:05.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:49:05.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:49:05.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:49:05.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:49:05.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:49:05.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:49:05.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:49:05.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:49:05.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:49:05.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:49:05.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:49:05.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:49:05.707 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:49:05.707 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:05 smithi005 ceph-mon[89904]: from='client.64372 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:49:05.707 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:05 smithi005 ceph-mon[89904]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:49:05.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:49:05.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:49:05.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:49:05.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:49:05.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:49:05.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:49:05.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:49:05.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:49:05.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:49:05.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:49:05.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:49:05.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:49:05.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:49:05.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:49:05.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:49:05.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:49:05.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:49:05.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:49:05.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:49:05.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:49:05.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:49:05.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:49:05.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:49:05.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:49:05.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:49:05.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:49:05.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:49:05.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:49:05.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: return future.result(timeout) 2024-03-22T05:49:05.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:49:05.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.__get_result() 2024-03-22T05:49:05.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:49:05.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: raise self._exception 2024-03-22T05:49:05.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:49:05.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:49:05.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:49:05.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:49:05.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:49:05.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:49:05.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:49:05.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:49:05.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:49:05.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:49:05.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:49:05.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:49:05.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:49:05.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:49:05.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:49:05.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:49:05.711 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:49:05.711 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:49:05.711 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:49:05.711 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:49:05.711 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:49:05.711 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:49:05.711 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:49:05.711 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:49:05.711 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:49:05.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:05 smithi005 ceph-mon[84705]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37 2024-03-22T05:49:05.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:05 smithi005 ceph-mon[84705]: from='client.54444 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:49:05.712 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:49:05.712 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:49:05.712 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:49:05.712 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:49:05.712 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:49:05.712 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:05 smithi005 ceph-mon[89904]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37 2024-03-22T05:49:05.712 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:05 smithi005 ceph-mon[89904]: from='client.54444 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:49:05.796 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:05 smithi186 ceph-mon[67903]: from='client.64372 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:49:05.796 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:05 smithi186 ceph-mon[67903]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:49:05.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:49:05.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:49:05.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:49:05.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:49:05.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:49:05.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:49:05.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:49:05.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:49:05.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:49:05.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:49:05.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:49:05.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:49:05.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:49:05.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:49:05.799 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:49:05.799 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:49:05.799 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:49:05.799 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:49:05.799 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:49:05.800 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:49:05.800 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:49:05.800 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:49:05.800 INFO:journalctl@ceph.mon.b.smithi186.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:49:05.800 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:49:05.800 INFO:journalctl@ceph.mon.b.smithi186.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:49:05.801 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:49:05.801 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:49:05.801 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:49:05.801 INFO:journalctl@ceph.mon.b.smithi186.stdout: return future.result(timeout) 2024-03-22T05:49:05.801 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:49:05.801 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.__get_result() 2024-03-22T05:49:05.802 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:49:05.802 INFO:journalctl@ceph.mon.b.smithi186.stdout: raise self._exception 2024-03-22T05:49:05.802 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:49:05.802 INFO:journalctl@ceph.mon.b.smithi186.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:49:05.802 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:49:05.802 INFO:journalctl@ceph.mon.b.smithi186.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:49:05.803 INFO:journalctl@ceph.mon.b.smithi186.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:49:05.803 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:49:05.803 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:49:05.803 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:49:05.803 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:49:05.804 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:49:05.804 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:49:05.804 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:49:05.804 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:49:05.804 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:49:05.804 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:49:05.805 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:49:05.805 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:49:05.805 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:49:05.805 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:49:05.805 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:49:05.805 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:49:05.805 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:49:05.806 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:49:05.806 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:49:05.806 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:49:05.806 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:05 smithi186 ceph-mon[67903]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37 2024-03-22T05:49:05.806 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:05 smithi186 ceph-mon[67903]: from='client.54444 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:49:06.058 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-03-22T05:49:06.058 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": { 2024-03-22T05:49:06.058 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev)": 3 2024-03-22T05:49:06.058 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:49:06.058 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr": { 2024-03-22T05:49:06.058 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev)": 2 2024-03-22T05:49:06.058 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:49:06.058 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": { 2024-03-22T05:49:06.058 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-03-22T05:49:06.059 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev)": 3 2024-03-22T05:49:06.059 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:49:06.059 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw": { 2024-03-22T05:49:06.059 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-22T05:49:06.059 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:49:06.059 INFO:teuthology.orchestra.run.smithi005.stdout: "overall": { 2024-03-22T05:49:06.059 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7, 2024-03-22T05:49:06.059 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev)": 8 2024-03-22T05:49:06.059 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-03-22T05:49:06.059 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-03-22T05:49:06.416 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-03-22T05:49:06.416 INFO:teuthology.orchestra.run.smithi005.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37", 2024-03-22T05:49:06.416 INFO:teuthology.orchestra.run.smithi005.stdout: "in_progress": true, 2024-03-22T05:49:06.416 INFO:teuthology.orchestra.run.smithi005.stdout: "which": "Upgrading daemons of type(s) crash,osd", 2024-03-22T05:49:06.416 INFO:teuthology.orchestra.run.smithi005.stdout: "services_complete": [], 2024-03-22T05:49:06.416 INFO:teuthology.orchestra.run.smithi005.stdout: "progress": "", 2024-03-22T05:49:06.416 INFO:teuthology.orchestra.run.smithi005.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37 image", 2024-03-22T05:49:06.416 INFO:teuthology.orchestra.run.smithi005.stdout: "is_paused": false 2024-03-22T05:49:06.416 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-03-22T05:49:06.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:06 smithi005 ceph-mon[84705]: from='client.54450 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:49:06.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:06 smithi005 ceph-mon[84705]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:49:06.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:06 smithi005 ceph-mon[84705]: from='client.? 172.21.15.5:0/1087343747' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:49:06.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:06 smithi005 ceph-mon[89904]: from='client.54450 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:49:06.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:06 smithi005 ceph-mon[89904]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:49:06.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:06 smithi005 ceph-mon[89904]: from='client.? 172.21.15.5:0/1087343747' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:49:06.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:06 smithi186 ceph-mon[67903]: from='client.54450 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:49:06.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:06 smithi186 ceph-mon[67903]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:49:06.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:06 smithi186 ceph-mon[67903]: from='client.? 172.21.15.5:0/1087343747' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:49:07.202 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:49:06] "GET /metrics HTTP/1.1" 200 38245 "" "Prometheus/2.43.0" 2024-03-22T05:49:08.445 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:08 smithi005 ceph-mon[89904]: from='client.54462 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:49:08.445 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:08 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:49:08.445 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:08 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:49:08.445 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:08 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:49:08.445 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:08 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:49:08.445 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:08 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:49:08.445 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:08 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:49:08.446 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:08 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:49:08.446 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:08 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:49:08.446 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:08 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:49:08.446 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:08 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-22T05:49:08.446 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:08 smithi005 ceph-mon[84705]: from='client.54462 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:49:08.446 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:08 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:49:08.446 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:08 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:49:08.446 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:08 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:49:08.446 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:08 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:49:08.446 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:08 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:49:08.446 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:08 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:49:08.447 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:08 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:49:08.447 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:08 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:49:08.447 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:08 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:49:08.447 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:08 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-22T05:49:08.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:08 smithi186 ceph-mon[67903]: from='client.54462 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:49:08.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:08 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:49:08.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:08 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:49:08.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:08 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:49:08.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:08 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:49:08.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:08 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:49:08.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:08 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:49:08.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:08 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:49:08.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:08 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:49:08.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:08 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:49:08.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:08 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-22T05:49:09.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:09 smithi005 ceph-mon[89904]: Upgrade: Target is version 19.0.0-2299-gea01217f (squid) 2024-03-22T05:49:09.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:09 smithi005 ceph-mon[89904]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, digests ['quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9'] 2024-03-22T05:49:09.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:09 smithi005 ceph-mon[89904]: Upgrade: Setting container_image for all mgr 2024-03-22T05:49:09.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:09 smithi005 ceph-mon[89904]: Upgrade: Setting container_image for all mon 2024-03-22T05:49:09.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:09 smithi005 ceph-mon[89904]: Upgrade: Setting container_image for all crash 2024-03-22T05:49:09.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:09 smithi005 ceph-mon[89904]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-22T05:49:09.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:09 smithi005 ceph-mon[89904]: Upgrade: osd.3 is safe to restart 2024-03-22T05:49:09.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:09 smithi005 ceph-mon[89904]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 609 B/s rd, 0 op/s 2024-03-22T05:49:09.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:09 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:49:09.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:09 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-22T05:49:09.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:09 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:49:09.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:09 smithi005 ceph-mon[84705]: Upgrade: Target is version 19.0.0-2299-gea01217f (squid) 2024-03-22T05:49:09.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:09 smithi005 ceph-mon[84705]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, digests ['quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9'] 2024-03-22T05:49:09.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:09 smithi005 ceph-mon[84705]: Upgrade: Setting container_image for all mgr 2024-03-22T05:49:09.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:09 smithi005 ceph-mon[84705]: Upgrade: Setting container_image for all mon 2024-03-22T05:49:09.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:09 smithi005 ceph-mon[84705]: Upgrade: Setting container_image for all crash 2024-03-22T05:49:09.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:09 smithi005 ceph-mon[84705]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-22T05:49:09.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:09 smithi005 ceph-mon[84705]: Upgrade: osd.3 is safe to restart 2024-03-22T05:49:09.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:09 smithi005 ceph-mon[84705]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 609 B/s rd, 0 op/s 2024-03-22T05:49:09.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:09 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:49:09.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:09 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-22T05:49:09.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:09 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:49:09.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:09 smithi186 ceph-mon[67903]: Upgrade: Target is version 19.0.0-2299-gea01217f (squid) 2024-03-22T05:49:09.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:09 smithi186 ceph-mon[67903]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, digests ['quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9'] 2024-03-22T05:49:09.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:09 smithi186 ceph-mon[67903]: Upgrade: Setting container_image for all mgr 2024-03-22T05:49:09.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:09 smithi186 ceph-mon[67903]: Upgrade: Setting container_image for all mon 2024-03-22T05:49:09.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:09 smithi186 ceph-mon[67903]: Upgrade: Setting container_image for all crash 2024-03-22T05:49:09.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:09 smithi186 ceph-mon[67903]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-22T05:49:09.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:09 smithi186 ceph-mon[67903]: Upgrade: osd.3 is safe to restart 2024-03-22T05:49:09.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:09 smithi186 ceph-mon[67903]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 609 B/s rd, 0 op/s 2024-03-22T05:49:09.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:09 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:49:09.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:09 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-22T05:49:09.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:09 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:49:10.363 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:10 smithi005 ceph-mon[84705]: Upgrade: Updating osd.3 2024-03-22T05:49:10.363 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:10 smithi005 ceph-mon[84705]: Deploying daemon osd.3 on smithi005 2024-03-22T05:49:10.363 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:10 smithi005 ceph-mon[89904]: Upgrade: Updating osd.3 2024-03-22T05:49:10.363 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:10 smithi005 ceph-mon[89904]: Deploying daemon osd.3 on smithi005 2024-03-22T05:49:10.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:10 smithi186 ceph-mon[67903]: Upgrade: Updating osd.3 2024-03-22T05:49:10.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:10 smithi186 ceph-mon[67903]: Deploying daemon osd.3 on smithi005 2024-03-22T05:49:11.430 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:11 smithi005 ceph-mon[84705]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:49:11.430 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:11 smithi005 ceph-mon[89904]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:49:11.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:11 smithi186 ceph-mon[67903]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:49:12.743 INFO:journalctl@ceph.osd.3.smithi005.stdout:Mar 22 05:49:12 smithi005 systemd[1]: Stopping Ceph osd.3 for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:49:13.202 INFO:journalctl@ceph.osd.3.smithi005.stdout:Mar 22 05:49:12 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-3[41223]: 2024-03-22T05:49:12.742+0000 7f3812d4b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T05:49:13.202 INFO:journalctl@ceph.osd.3.smithi005.stdout:Mar 22 05:49:12 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-3[41223]: 2024-03-22T05:49:12.742+0000 7f3812d4b700 -1 osd.3 120 *** Got signal Terminated *** 2024-03-22T05:49:13.202 INFO:journalctl@ceph.osd.3.smithi005.stdout:Mar 22 05:49:12 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-3[41223]: 2024-03-22T05:49:12.742+0000 7f3812d4b700 -1 osd.3 120 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-22T05:49:13.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:13 smithi186 ceph-mon[67903]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:49:13.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:13 smithi186 ceph-mon[67903]: osd.3 marked itself down and dead 2024-03-22T05:49:13.644 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:13 smithi005 ceph-mon[84705]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:49:13.644 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:13 smithi005 ceph-mon[84705]: osd.3 marked itself down and dead 2024-03-22T05:49:13.644 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:13 smithi005 ceph-mon[89904]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:49:13.645 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:13 smithi005 ceph-mon[89904]: osd.3 marked itself down and dead 2024-03-22T05:49:13.935 INFO:journalctl@ceph.osd.3.smithi005.stdout:Mar 22 05:49:13 smithi005 podman[116805]: 2024-03-22 05:49:13.659839976 +0000 UTC m=+1.041486049 container died b25432a7239b672e5e1ba0eb5e2d82840a4fd1705eb26ad8058f7cacd203c29a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-3, io.k8s.display-name=CentOS Stream 8, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, GIT_BRANCH=HEAD, release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, io.openshift.expose-services=, ceph=True, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc.) 2024-03-22T05:49:14.202 INFO:journalctl@ceph.osd.3.smithi005.stdout:Mar 22 05:49:14 smithi005 podman[116805]: 2024-03-22 05:49:14.025892284 +0000 UTC m=+1.407538357 container cleanup b25432a7239b672e5e1ba0eb5e2d82840a4fd1705eb26ad8058f7cacd203c29a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-3, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, distribution-scope=public, io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, name=centos-stream, com.redhat.component=centos-stream-container, version=8, vcs-type=git, io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64) 2024-03-22T05:49:14.203 INFO:journalctl@ceph.osd.3.smithi005.stdout:Mar 22 05:49:14 smithi005 bash[116805]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-3 2024-03-22T05:49:14.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:14 smithi186 ceph-mon[67903]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-22T05:49:14.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:14 smithi186 ceph-mon[67903]: osdmap e121: 8 total, 7 up, 8 in 2024-03-22T05:49:14.684 INFO:journalctl@ceph.osd.3.smithi005.stdout:Mar 22 05:49:14 smithi005 podman[116834]: 2024-03-22 05:49:14.327381808 +0000 UTC m=+0.664675044 container remove b25432a7239b672e5e1ba0eb5e2d82840a4fd1705eb26ad8058f7cacd203c29a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-3, build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, io.openshift.tags=base centos centos-stream, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, io.openshift.expose-services=, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , RELEASE=HEAD, vcs-type=git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, distribution-scope=public) 2024-03-22T05:49:14.685 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:14 smithi005 ceph-mon[84705]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-22T05:49:14.685 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:14 smithi005 ceph-mon[84705]: osdmap e121: 8 total, 7 up, 8 in 2024-03-22T05:49:14.685 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:14 smithi005 ceph-mon[89904]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-22T05:49:14.685 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:14 smithi005 ceph-mon[89904]: osdmap e121: 8 total, 7 up, 8 in 2024-03-22T05:49:14.947 INFO:journalctl@ceph.osd.3.smithi005.stdout:Mar 22 05:49:14 smithi005 podman[116929]: 2024-03-22 05:49:14.588448236 +0000 UTC m=+0.018911006 image pull aa461e7c5e7f322b169c789b389d594b994ea7ffcbc9e24e563a3ec9b69bc3c1 quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9 2024-03-22T05:49:14.947 INFO:journalctl@ceph.osd.3.smithi005.stdout:Mar 22 05:49:14 smithi005 podman[116929]: 2024-03-22 05:49:14.711673593 +0000 UTC m=+0.142136363 container create 3b54b4687cd4eec06a0c1a332a56f384a5ddd58909846f8b9f42168acd549e79 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-3-deactivate, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, maintainer=Guillaume Abrioux ) 2024-03-22T05:49:15.202 INFO:journalctl@ceph.osd.3.smithi005.stdout:Mar 22 05:49:14 smithi005 podman[116929]: 2024-03-22 05:49:14.94820219 +0000 UTC m=+0.378664959 container init 3b54b4687cd4eec06a0c1a332a56f384a5ddd58909846f8b9f42168acd549e79 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, org.label-schema.vendor=CentOS) 2024-03-22T05:49:15.202 INFO:journalctl@ceph.osd.3.smithi005.stdout:Mar 22 05:49:14 smithi005 podman[116929]: 2024-03-22 05:49:14.951324535 +0000 UTC m=+0.381787307 container start 3b54b4687cd4eec06a0c1a332a56f384a5ddd58909846f8b9f42168acd549e79 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-3-deactivate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217) 2024-03-22T05:49:15.202 INFO:journalctl@ceph.osd.3.smithi005.stdout:Mar 22 05:49:15 smithi005 podman[116929]: 2024-03-22 05:49:15.03969156 +0000 UTC m=+0.470154343 container attach 3b54b4687cd4eec06a0c1a332a56f384a5ddd58909846f8b9f42168acd549e79 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-3-deactivate, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-22T05:49:15.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:15 smithi186 ceph-mon[67903]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:49:15.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:15 smithi186 ceph-mon[67903]: pgmap v170: 161 pgs: 39 peering, 122 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:49:15.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:15 smithi186 ceph-mon[67903]: Health check failed: Reduced data availability: 9 pgs inactive, 14 pgs peering (PG_AVAILABILITY) 2024-03-22T05:49:15.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:15 smithi186 ceph-mon[67903]: osdmap e122: 8 total, 7 up, 8 in 2024-03-22T05:49:15.550 INFO:journalctl@ceph.osd.3.smithi005.stdout:Mar 22 05:49:15 smithi005 podman[116929]: 2024-03-22 05:49:15.262066473 +0000 UTC m=+0.692529245 container died 3b54b4687cd4eec06a0c1a332a56f384a5ddd58909846f8b9f42168acd549e79 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-22T05:49:15.551 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:15 smithi005 ceph-mon[84705]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:49:15.551 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:15 smithi005 ceph-mon[84705]: pgmap v170: 161 pgs: 39 peering, 122 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:49:15.551 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:15 smithi005 ceph-mon[84705]: Health check failed: Reduced data availability: 9 pgs inactive, 14 pgs peering (PG_AVAILABILITY) 2024-03-22T05:49:15.551 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:15 smithi005 ceph-mon[84705]: osdmap e122: 8 total, 7 up, 8 in 2024-03-22T05:49:15.551 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:15 smithi005 ceph-mon[89904]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:49:15.551 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:15 smithi005 ceph-mon[89904]: pgmap v170: 161 pgs: 39 peering, 122 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:49:15.551 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:15 smithi005 ceph-mon[89904]: Health check failed: Reduced data availability: 9 pgs inactive, 14 pgs peering (PG_AVAILABILITY) 2024-03-22T05:49:15.551 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:15 smithi005 ceph-mon[89904]: osdmap e122: 8 total, 7 up, 8 in 2024-03-22T05:49:16.202 INFO:journalctl@ceph.osd.3.smithi005.stdout:Mar 22 05:49:15 smithi005 podman[116973]: 2024-03-22 05:49:15.927498984 +0000 UTC m=+0.660528651 container remove 3b54b4687cd4eec06a0c1a332a56f384a5ddd58909846f8b9f42168acd549e79 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-3-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-22T05:49:16.202 INFO:journalctl@ceph.osd.3.smithi005.stdout:Mar 22 05:49:15 smithi005 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@osd.3.service: Deactivated successfully. 2024-03-22T05:49:16.203 INFO:journalctl@ceph.osd.3.smithi005.stdout:Mar 22 05:49:15 smithi005 systemd[1]: Stopped Ceph osd.3 for f59f36e2-e80a-11ee-95cd-87774f69a715. 2024-03-22T05:49:16.203 INFO:journalctl@ceph.osd.3.smithi005.stdout:Mar 22 05:49:15 smithi005 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@osd.3.service: Consumed 17.419s CPU time. 2024-03-22T05:49:16.203 INFO:journalctl@ceph.osd.3.smithi005.stdout:Mar 22 05:49:16 smithi005 systemd[1]: Starting Ceph osd.3 for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:49:16.647 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:16 smithi005 ceph-mon[84705]: Health check failed: Degraded data redundancy: 24/723 objects degraded (3.320%), 5 pgs degraded (PG_DEGRADED) 2024-03-22T05:49:16.647 INFO:journalctl@ceph.osd.3.smithi005.stdout:Mar 22 05:49:16 smithi005 podman[117072]: 2024-03-22 05:49:16.282382516 +0000 UTC m=+0.017876891 image pull aa461e7c5e7f322b169c789b389d594b994ea7ffcbc9e24e563a3ec9b69bc3c1 quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9 2024-03-22T05:49:16.648 INFO:journalctl@ceph.osd.3.smithi005.stdout:Mar 22 05:49:16 smithi005 podman[117072]: 2024-03-22 05:49:16.42024279 +0000 UTC m=+0.155737193 container create 5293f53792666f63db260a539881dc08db297cdfd0801b41f2e99f4a9a14a872 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-3-activate, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, maintainer=Guillaume Abrioux ) 2024-03-22T05:49:16.648 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:16 smithi005 ceph-mon[89904]: Health check failed: Degraded data redundancy: 24/723 objects degraded (3.320%), 5 pgs degraded (PG_DEGRADED) 2024-03-22T05:49:16.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:16 smithi186 ceph-mon[67903]: Health check failed: Degraded data redundancy: 24/723 objects degraded (3.320%), 5 pgs degraded (PG_DEGRADED) 2024-03-22T05:49:16.952 INFO:journalctl@ceph.osd.3.smithi005.stdout:Mar 22 05:49:16 smithi005 podman[117072]: 2024-03-22 05:49:16.648165327 +0000 UTC m=+0.383659689 container init 5293f53792666f63db260a539881dc08db297cdfd0801b41f2e99f4a9a14a872 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-3-activate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-22T05:49:16.952 INFO:journalctl@ceph.osd.3.smithi005.stdout:Mar 22 05:49:16 smithi005 podman[117072]: 2024-03-22 05:49:16.651326918 +0000 UTC m=+0.386821279 container start 5293f53792666f63db260a539881dc08db297cdfd0801b41f2e99f4a9a14a872 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-3-activate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-22T05:49:16.952 INFO:journalctl@ceph.osd.3.smithi005.stdout:Mar 22 05:49:16 smithi005 podman[117072]: 2024-03-22 05:49:16.756592758 +0000 UTC m=+0.492087136 container attach 5293f53792666f63db260a539881dc08db297cdfd0801b41f2e99f4a9a14a872 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-3-activate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217) 2024-03-22T05:49:17.302 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:16 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:49:16] "GET /metrics HTTP/1.1" 200 38242 "" "Prometheus/2.43.0" 2024-03-22T05:49:17.302 INFO:journalctl@ceph.osd.3.smithi005.stdout:Mar 22 05:49:17 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-3-activate[117088]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-22T05:49:17.303 INFO:journalctl@ceph.osd.3.smithi005.stdout:Mar 22 05:49:17 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-3-activate[117088]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-03-22T05:49:17.303 INFO:journalctl@ceph.osd.3.smithi005.stdout:Mar 22 05:49:17 smithi005 bash[117072]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-22T05:49:17.303 INFO:journalctl@ceph.osd.3.smithi005.stdout:Mar 22 05:49:17 smithi005 bash[117072]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-03-22T05:49:17.303 INFO:journalctl@ceph.osd.3.smithi005.stdout:Mar 22 05:49:17 smithi005 bash[117072]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-03-22T05:49:17.303 INFO:journalctl@ceph.osd.3.smithi005.stdout:Mar 22 05:49:17 smithi005 bash[117072]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-03-22T05:49:17.303 INFO:journalctl@ceph.osd.3.smithi005.stdout:Mar 22 05:49:17 smithi005 bash[117072]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-03-22T05:49:17.303 INFO:journalctl@ceph.osd.3.smithi005.stdout:Mar 22 05:49:17 smithi005 bash[117072]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-22T05:49:17.303 INFO:journalctl@ceph.osd.3.smithi005.stdout:Mar 22 05:49:17 smithi005 bash[117072]: --> ceph-volume raw activate successful for osd ID: 3 2024-03-22T05:49:17.303 INFO:journalctl@ceph.osd.3.smithi005.stdout:Mar 22 05:49:17 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-3-activate[117088]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-03-22T05:49:17.303 INFO:journalctl@ceph.osd.3.smithi005.stdout:Mar 22 05:49:17 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-3-activate[117088]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-03-22T05:49:17.303 INFO:journalctl@ceph.osd.3.smithi005.stdout:Mar 22 05:49:17 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-3-activate[117088]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-03-22T05:49:17.303 INFO:journalctl@ceph.osd.3.smithi005.stdout:Mar 22 05:49:17 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-3-activate[117088]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-22T05:49:17.303 INFO:journalctl@ceph.osd.3.smithi005.stdout:Mar 22 05:49:17 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-3-activate[117088]: --> ceph-volume raw activate successful for osd ID: 3 2024-03-22T05:49:17.303 INFO:journalctl@ceph.osd.3.smithi005.stdout:Mar 22 05:49:17 smithi005 podman[117072]: 2024-03-22 05:49:17.124627268 +0000 UTC m=+0.860121641 container died 5293f53792666f63db260a539881dc08db297cdfd0801b41f2e99f4a9a14a872 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-3-activate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-22T05:49:17.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:17 smithi005 ceph-mon[89904]: pgmap v172: 161 pgs: 6 stale+active+clean, 5 active+undersized+degraded, 39 peering, 8 active+undersized, 103 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 24/723 objects degraded (3.320%) 2024-03-22T05:49:17.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:17 smithi005 ceph-mon[84705]: pgmap v172: 161 pgs: 6 stale+active+clean, 5 active+undersized+degraded, 39 peering, 8 active+undersized, 103 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 24/723 objects degraded (3.320%) 2024-03-22T05:49:17.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:17 smithi186 ceph-mon[67903]: pgmap v172: 161 pgs: 6 stale+active+clean, 5 active+undersized+degraded, 39 peering, 8 active+undersized, 103 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 24/723 objects degraded (3.320%) 2024-03-22T05:49:18.149 INFO:journalctl@ceph.osd.3.smithi005.stdout:Mar 22 05:49:17 smithi005 podman[117207]: 2024-03-22 05:49:17.821520046 +0000 UTC m=+0.692249478 container remove 5293f53792666f63db260a539881dc08db297cdfd0801b41f2e99f4a9a14a872 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-3-activate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-22T05:49:18.150 INFO:journalctl@ceph.osd.3.smithi005.stdout:Mar 22 05:49:18 smithi005 podman[117272]: 2024-03-22 05:49:17.944434083 +0000 UTC m=+0.018770847 image pull aa461e7c5e7f322b169c789b389d594b994ea7ffcbc9e24e563a3ec9b69bc3c1 quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9 2024-03-22T05:49:18.150 INFO:journalctl@ceph.osd.3.smithi005.stdout:Mar 22 05:49:18 smithi005 podman[117272]: 2024-03-22 05:49:18.053338997 +0000 UTC m=+0.127675760 container create d31a01a0230c2a13213540f3096e04da54ea6bae5316cce09f70ab08e767dbc0 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-3, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=) 2024-03-22T05:49:18.452 INFO:journalctl@ceph.osd.3.smithi005.stdout:Mar 22 05:49:18 smithi005 podman[117272]: 2024-03-22 05:49:18.272868039 +0000 UTC m=+0.347204812 container init d31a01a0230c2a13213540f3096e04da54ea6bae5316cce09f70ab08e767dbc0 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-3, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-22T05:49:18.452 INFO:journalctl@ceph.osd.3.smithi005.stdout:Mar 22 05:49:18 smithi005 podman[117272]: 2024-03-22 05:49:18.275897142 +0000 UTC m=+0.350233908 container start d31a01a0230c2a13213540f3096e04da54ea6bae5316cce09f70ab08e767dbc0 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-3, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, io.buildah.version=1.33.5) 2024-03-22T05:49:18.452 INFO:journalctl@ceph.osd.3.smithi005.stdout:Mar 22 05:49:18 smithi005 ceph-osd[117291]: -- 172.21.15.5:0/3242020923 <== mon.0 v2:172.21.15.5:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5635a3fd7680 con 0x5635a3fa1400 2024-03-22T05:49:18.452 INFO:journalctl@ceph.osd.3.smithi005.stdout:Mar 22 05:49:18 smithi005 bash[117272]: d31a01a0230c2a13213540f3096e04da54ea6bae5316cce09f70ab08e767dbc0 2024-03-22T05:49:18.452 INFO:journalctl@ceph.osd.3.smithi005.stdout:Mar 22 05:49:18 smithi005 systemd[1]: Started Ceph osd.3 for f59f36e2-e80a-11ee-95cd-87774f69a715. 2024-03-22T05:49:19.202 INFO:journalctl@ceph.osd.3.smithi005.stdout:Mar 22 05:49:18 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-3[117287]: 2024-03-22T05:49:18.828+0000 7f1ea32d37c0 -1 Falling back to public interface 2024-03-22T05:49:19.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:19 smithi005 ceph-mon[84705]: pgmap v173: 161 pgs: 11 active+undersized+degraded, 39 peering, 16 active+undersized, 95 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 51/723 objects degraded (7.054%) 2024-03-22T05:49:19.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:19 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:49:19.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:19 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:49:19.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:19 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:49:19.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:19 smithi005 ceph-mon[89904]: pgmap v173: 161 pgs: 11 active+undersized+degraded, 39 peering, 16 active+undersized, 95 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 51/723 objects degraded (7.054%) 2024-03-22T05:49:19.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:19 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:49:19.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:19 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:49:19.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:19 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:49:19.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:19 smithi186 ceph-mon[67903]: pgmap v173: 161 pgs: 11 active+undersized+degraded, 39 peering, 16 active+undersized, 95 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 51/723 objects degraded (7.054%) 2024-03-22T05:49:19.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:19 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:49:19.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:19 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:49:19.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:19 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:49:20.428 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:20 smithi005 ceph-mon[84705]: pgmap v174: 161 pgs: 24 active+undersized+degraded, 42 active+undersized, 95 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 93/723 objects degraded (12.863%) 2024-03-22T05:49:20.429 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:20 smithi005 ceph-mon[84705]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 9 pgs inactive, 14 pgs peering) 2024-03-22T05:49:20.429 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:20 smithi005 ceph-mon[89904]: pgmap v174: 161 pgs: 24 active+undersized+degraded, 42 active+undersized, 95 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 93/723 objects degraded (12.863%) 2024-03-22T05:49:20.429 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:20 smithi005 ceph-mon[89904]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 9 pgs inactive, 14 pgs peering) 2024-03-22T05:49:20.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:20 smithi186 ceph-mon[67903]: pgmap v174: 161 pgs: 24 active+undersized+degraded, 42 active+undersized, 95 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 93/723 objects degraded (12.863%) 2024-03-22T05:49:20.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:20 smithi186 ceph-mon[67903]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 9 pgs inactive, 14 pgs peering) 2024-03-22T05:49:21.820 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:21 smithi005 ceph-mon[84705]: from='osd.3 [v2:172.21.15.5:6826/2284992870,v1:172.21.15.5:6827/2284992870]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-22T05:49:21.820 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:21 smithi005 ceph-mon[84705]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-22T05:49:21.820 INFO:journalctl@ceph.osd.3.smithi005.stdout:Mar 22 05:49:21 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-3[117287]: 2024-03-22T05:49:21.562+0000 7f1ea32d37c0 -1 osd.3 0 read_superblock omap replica is missing. 2024-03-22T05:49:21.820 INFO:journalctl@ceph.osd.3.smithi005.stdout:Mar 22 05:49:21 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-3[117287]: 2024-03-22T05:49:21.590+0000 7f1ea32d37c0 -1 osd.3 120 log_to_monitors true 2024-03-22T05:49:21.820 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:21 smithi005 ceph-mon[89904]: from='osd.3 [v2:172.21.15.5:6826/2284992870,v1:172.21.15.5:6827/2284992870]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-22T05:49:21.821 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:21 smithi005 ceph-mon[89904]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-22T05:49:22.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:21 smithi186 ceph-mon[67903]: from='osd.3 [v2:172.21.15.5:6826/2284992870,v1:172.21.15.5:6827/2284992870]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-22T05:49:22.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:21 smithi186 ceph-mon[67903]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-22T05:49:22.202 INFO:journalctl@ceph.osd.3.smithi005.stdout:Mar 22 05:49:21 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-3[117287]: 2024-03-22T05:49:21.932+0000 7f1e90723700 -1 osd.3 120 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-22T05:49:22.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:22 smithi005 ceph-mon[89904]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-22T05:49:22.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:22 smithi005 ceph-mon[89904]: osdmap e123: 8 total, 7 up, 8 in 2024-03-22T05:49:22.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:22 smithi005 ceph-mon[89904]: from='osd.3 [v2:172.21.15.5:6826/2284992870,v1:172.21.15.5:6827/2284992870]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-03-22T05:49:22.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:22 smithi005 ceph-mon[89904]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-03-22T05:49:22.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:22 smithi005 ceph-mon[89904]: pgmap v176: 161 pgs: 24 active+undersized+degraded, 42 active+undersized, 95 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s; 93/723 objects degraded (12.863%) 2024-03-22T05:49:22.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:22 smithi005 ceph-mon[89904]: from='osd.3 ' entity='osd.3' 2024-03-22T05:49:22.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:22 smithi005 ceph-mon[89904]: Health check update: Degraded data redundancy: 93/723 objects degraded (12.863%), 24 pgs degraded (PG_DEGRADED) 2024-03-22T05:49:22.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:22 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:49:22.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:22 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:49:22.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:22 smithi005 ceph-mon[84705]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-22T05:49:22.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:22 smithi005 ceph-mon[84705]: osdmap e123: 8 total, 7 up, 8 in 2024-03-22T05:49:22.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:22 smithi005 ceph-mon[84705]: from='osd.3 [v2:172.21.15.5:6826/2284992870,v1:172.21.15.5:6827/2284992870]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-03-22T05:49:22.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:22 smithi005 ceph-mon[84705]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-03-22T05:49:22.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:22 smithi005 ceph-mon[84705]: pgmap v176: 161 pgs: 24 active+undersized+degraded, 42 active+undersized, 95 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s; 93/723 objects degraded (12.863%) 2024-03-22T05:49:22.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:22 smithi005 ceph-mon[84705]: from='osd.3 ' entity='osd.3' 2024-03-22T05:49:22.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:22 smithi005 ceph-mon[84705]: Health check update: Degraded data redundancy: 93/723 objects degraded (12.863%), 24 pgs degraded (PG_DEGRADED) 2024-03-22T05:49:22.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:22 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:49:22.954 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:22 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:49:23.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:22 smithi186 ceph-mon[67903]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-22T05:49:23.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:22 smithi186 ceph-mon[67903]: osdmap e123: 8 total, 7 up, 8 in 2024-03-22T05:49:23.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:22 smithi186 ceph-mon[67903]: from='osd.3 [v2:172.21.15.5:6826/2284992870,v1:172.21.15.5:6827/2284992870]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-03-22T05:49:23.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:22 smithi186 ceph-mon[67903]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi005", "root=default"]}]: dispatch 2024-03-22T05:49:23.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:22 smithi186 ceph-mon[67903]: pgmap v176: 161 pgs: 24 active+undersized+degraded, 42 active+undersized, 95 active+clean; 585 KiB data, 168 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s; 93/723 objects degraded (12.863%) 2024-03-22T05:49:23.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:22 smithi186 ceph-mon[67903]: from='osd.3 ' entity='osd.3' 2024-03-22T05:49:23.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:22 smithi186 ceph-mon[67903]: Health check update: Degraded data redundancy: 93/723 objects degraded (12.863%), 24 pgs degraded (PG_DEGRADED) 2024-03-22T05:49:23.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:22 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:49:23.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:22 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:49:23.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:23 smithi005 ceph-mon[84705]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-22T05:49:23.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:23 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:49:23.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:23 smithi005 ceph-mon[84705]: osd.3 [v2:172.21.15.5:6826/2284992870,v1:172.21.15.5:6827/2284992870] boot 2024-03-22T05:49:23.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:23 smithi005 ceph-mon[84705]: osdmap e124: 8 total, 8 up, 8 in 2024-03-22T05:49:23.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:23 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T05:49:23.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:23 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:49:23.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:23 smithi005 ceph-mon[89904]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-22T05:49:23.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:23 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:49:23.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:23 smithi005 ceph-mon[89904]: osd.3 [v2:172.21.15.5:6826/2284992870,v1:172.21.15.5:6827/2284992870] boot 2024-03-22T05:49:23.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:23 smithi005 ceph-mon[89904]: osdmap e124: 8 total, 8 up, 8 in 2024-03-22T05:49:23.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:23 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T05:49:23.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:23 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:49:24.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:23 smithi186 ceph-mon[67903]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-22T05:49:24.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:23 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:49:24.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:23 smithi186 ceph-mon[67903]: osd.3 [v2:172.21.15.5:6826/2284992870,v1:172.21.15.5:6827/2284992870] boot 2024-03-22T05:49:24.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:23 smithi186 ceph-mon[67903]: osdmap e124: 8 total, 8 up, 8 in 2024-03-22T05:49:24.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:23 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-22T05:49:24.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:23 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:49:25.087 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:24 smithi005 ceph-mon[89904]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:49:25.087 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:24 smithi005 ceph-mon[89904]: pgmap v178: 161 pgs: 23 peering, 16 active+undersized+degraded, 27 active+undersized, 95 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s; 71/723 objects degraded (9.820%) 2024-03-22T05:49:25.087 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:24 smithi005 ceph-mon[89904]: osdmap e125: 8 total, 8 up, 8 in 2024-03-22T05:49:25.087 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:24 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:49:25.087 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:24 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:49:25.087 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:24 smithi005 ceph-mon[84705]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:49:25.087 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:24 smithi005 ceph-mon[84705]: pgmap v178: 161 pgs: 23 peering, 16 active+undersized+degraded, 27 active+undersized, 95 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s; 71/723 objects degraded (9.820%) 2024-03-22T05:49:25.087 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:24 smithi005 ceph-mon[84705]: osdmap e125: 8 total, 8 up, 8 in 2024-03-22T05:49:25.088 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:24 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:49:25.088 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:24 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:49:25.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:24 smithi186 ceph-mon[67903]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:49:25.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:24 smithi186 ceph-mon[67903]: pgmap v178: 161 pgs: 23 peering, 16 active+undersized+degraded, 27 active+undersized, 95 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s; 71/723 objects degraded (9.820%) 2024-03-22T05:49:25.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:24 smithi186 ceph-mon[67903]: osdmap e125: 8 total, 8 up, 8 in 2024-03-22T05:49:25.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:24 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:49:25.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:24 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:49:27.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:26 smithi005 ceph-mon[84705]: pgmap v180: 161 pgs: 23 peering, 13 active+undersized+degraded, 22 active+undersized, 103 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 57/723 objects degraded (7.884%) 2024-03-22T05:49:27.203 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:26 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:49:26] "GET /metrics HTTP/1.1" 200 38094 "" "Prometheus/2.43.0" 2024-03-22T05:49:27.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:26 smithi005 ceph-mon[89904]: pgmap v180: 161 pgs: 23 peering, 13 active+undersized+degraded, 22 active+undersized, 103 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 57/723 objects degraded (7.884%) 2024-03-22T05:49:27.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:26 smithi186 ceph-mon[67903]: pgmap v180: 161 pgs: 23 peering, 13 active+undersized+degraded, 22 active+undersized, 103 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 57/723 objects degraded (7.884%) 2024-03-22T05:49:28.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:27 smithi005 ceph-mon[84705]: Health check update: Degraded data redundancy: 57/723 objects degraded (7.884%), 13 pgs degraded (PG_DEGRADED) 2024-03-22T05:49:28.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:27 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:49:28.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:27 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:49:28.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:27 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:49:28.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:27 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:49:28.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:27 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:49:28.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:27 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:49:28.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:27 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:49:28.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:27 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:49:28.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:27 smithi005 ceph-mon[89904]: Health check update: Degraded data redundancy: 57/723 objects degraded (7.884%), 13 pgs degraded (PG_DEGRADED) 2024-03-22T05:49:28.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:27 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:49:28.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:27 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:49:28.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:27 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:49:28.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:27 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:49:28.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:27 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:49:28.204 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:27 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:49:28.204 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:27 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:49:28.204 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:27 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:49:28.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:27 smithi186 ceph-mon[67903]: Health check update: Degraded data redundancy: 57/723 objects degraded (7.884%), 13 pgs degraded (PG_DEGRADED) 2024-03-22T05:49:28.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:27 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:49:28.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:27 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:49:28.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:27 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:49:28.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:27 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:49:28.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:27 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:49:28.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:27 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:49:28.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:27 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:49:28.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:27 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:49:29.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:28 smithi005 ceph-mon[89904]: pgmap v181: 161 pgs: 23 peering, 138 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:49:29.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:28 smithi005 ceph-mon[89904]: pgmap v182: 161 pgs: 23 peering, 138 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:49:29.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:28 smithi005 ceph-mon[89904]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:49:29.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:28 smithi005 ceph-mon[89904]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:49:29.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:28 smithi005 ceph-mon[89904]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 57/723 objects degraded (7.884%), 13 pgs degraded) 2024-03-22T05:49:29.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:28 smithi005 ceph-mon[84705]: pgmap v181: 161 pgs: 23 peering, 138 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:49:29.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:28 smithi005 ceph-mon[84705]: pgmap v182: 161 pgs: 23 peering, 138 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:49:29.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:28 smithi005 ceph-mon[84705]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:49:29.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:28 smithi005 ceph-mon[84705]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:49:29.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:28 smithi005 ceph-mon[84705]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 57/723 objects degraded (7.884%), 13 pgs degraded) 2024-03-22T05:49:29.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:28 smithi186 ceph-mon[67903]: pgmap v181: 161 pgs: 23 peering, 138 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:49:29.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:28 smithi186 ceph-mon[67903]: pgmap v182: 161 pgs: 23 peering, 138 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:49:29.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:28 smithi186 ceph-mon[67903]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:49:29.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:28 smithi186 ceph-mon[67903]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:49:29.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:28 smithi186 ceph-mon[67903]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 57/723 objects degraded (7.884%), 13 pgs degraded) 2024-03-22T05:49:29.943 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:29 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:49:29.643+0000 7f7f0ecc9700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:49:29.943 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:29 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:49:29.943 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:29 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:49:29.943 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:29 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:49:29.943 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:29 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:49:29.944 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:29 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:49:29.944 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:29 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:49:29.944 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:29 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:49:29.944 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:29 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:49:29.944 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:29 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:49:29.944 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:29 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:49:29.944 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:29 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:49:29.944 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:29 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:49:29.944 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:29 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:49:29.944 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:29 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:49:29.944 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:29 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:49:29.944 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:29 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:49:29.944 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:29 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:49:29.944 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:29 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:49:29.945 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:29 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:49:29.945 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:29 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:49:29.945 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:29 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:49:29.945 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:29 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:49:29.945 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:29 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:49:29.945 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:29 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:49:29.946 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:29 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:49:29.946 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:29 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:49:29.946 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:29 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:49:29.946 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:29 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:49:29.946 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:29 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return future.result(timeout) 2024-03-22T05:49:29.946 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:29 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:49:29.946 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:29 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.__get_result() 2024-03-22T05:49:29.946 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:29 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:49:29.946 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:29 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: raise self._exception 2024-03-22T05:49:29.946 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:29 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:49:29.946 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:29 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: config_blobs=daemon_spec.final_config, 2024-03-22T05:49:29.946 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:29 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:49:29.946 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:29 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:49:29.946 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:29 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:49:29.947 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:29 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:49:29.947 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:29 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:49:29.947 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:29 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:49:29.947 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:29 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:49:29.947 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:29 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:49:29.947 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:29 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:49:29.947 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:29 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:49:29.947 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:29 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:49:29.947 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:29 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:49:29.947 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:29 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:49:29.947 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:29 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:49:29.947 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:29 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:49:29.947 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:29 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:49:29.947 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:29 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:49:29.947 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:29 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:49:29.948 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:29 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:49:29.948 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:29 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:49:29.948 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:29 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:49:29.948 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:29 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:49:29.948 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:29 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:49:29.948 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:29 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:49:29.657+0000 7f7f3e571700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-03-22T05:49:30.195 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:29 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:49:30.195 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:29 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:49:30.195 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:29 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:49:30.195 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:29 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:49:30.195 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:29 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-22T05:49:30.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:29 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:49:30.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:29 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:49:30.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:29 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:49:30.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:29 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:49:30.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:29 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-22T05:49:30.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:29 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:49:30.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:29 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:49:30.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:29 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:49:30.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:29 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:49:30.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:29 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-22T05:49:31.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:30 smithi005 ceph-mon[84705]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:49:31.203 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:49:31.203 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:49:31.203 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:49:31.203 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:49:31.203 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:49:31.203 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:49:31.203 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:49:31.204 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:49:31.204 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:49:31.204 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:49:31.204 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:49:31.204 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:49:31.204 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:49:31.204 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:49:31.204 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:49:31.204 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:49:31.204 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:49:31.204 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:49:31.204 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:49:31.204 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:49:31.204 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:49:31.204 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:49:31.205 INFO:journalctl@ceph.mon.a.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:49:31.205 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:49:31.205 INFO:journalctl@ceph.mon.a.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:49:31.205 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:49:31.205 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:49:31.205 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:49:31.205 INFO:journalctl@ceph.mon.a.smithi005.stdout: return future.result(timeout) 2024-03-22T05:49:31.205 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:49:31.205 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.__get_result() 2024-03-22T05:49:31.205 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:49:31.205 INFO:journalctl@ceph.mon.a.smithi005.stdout: raise self._exception 2024-03-22T05:49:31.205 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:49:31.205 INFO:journalctl@ceph.mon.a.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:49:31.205 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:49:31.205 INFO:journalctl@ceph.mon.a.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:49:31.206 INFO:journalctl@ceph.mon.a.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:49:31.206 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:49:31.206 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:49:31.206 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:49:31.206 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:49:31.206 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:49:31.206 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:49:31.206 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:49:31.206 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:49:31.206 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:49:31.206 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:49:31.206 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:49:31.206 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:49:31.206 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:49:31.207 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:49:31.207 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:49:31.207 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:30 smithi005 ceph-mon[89904]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:49:31.207 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:49:31.207 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:49:31.207 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:49:31.207 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:49:31.207 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:49:31.207 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:49:31.207 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:49:31.207 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:49:31.208 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:49:31.208 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:49:31.208 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:49:31.208 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:49:31.208 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:49:31.208 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:49:31.208 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:49:31.208 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:49:31.208 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:49:31.208 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:49:31.208 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:49:31.208 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:49:31.208 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:49:31.208 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:49:31.208 INFO:journalctl@ceph.mon.c.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:49:31.208 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:49:31.209 INFO:journalctl@ceph.mon.c.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:49:31.209 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:49:31.209 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:49:31.209 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:49:31.209 INFO:journalctl@ceph.mon.c.smithi005.stdout: return future.result(timeout) 2024-03-22T05:49:31.209 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:49:31.209 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.__get_result() 2024-03-22T05:49:31.209 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:49:31.209 INFO:journalctl@ceph.mon.c.smithi005.stdout: raise self._exception 2024-03-22T05:49:31.209 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:49:31.209 INFO:journalctl@ceph.mon.c.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:49:31.209 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:49:31.209 INFO:journalctl@ceph.mon.c.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:49:31.209 INFO:journalctl@ceph.mon.c.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:49:31.209 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:49:31.209 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:49:31.210 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:49:31.210 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:49:31.210 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:49:31.210 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:49:31.210 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:49:31.210 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:49:31.210 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:49:31.210 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:49:31.210 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:49:31.210 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:49:31.210 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:49:31.210 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:49:31.210 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:49:31.210 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:49:31.211 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:49:31.211 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:49:31.211 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:49:31.211 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:49:31.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:30 smithi005 ceph-mon[84705]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-22T05:49:31.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:30 smithi005 ceph-mon[84705]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-03-22T05:49:31.211 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:30 smithi005 ceph-mon[84705]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:49:31.211 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:49:31.211 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:49:31.211 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:49:31.211 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:49:31.211 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:49:31.211 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:30 smithi005 ceph-mon[89904]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-22T05:49:31.211 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:30 smithi005 ceph-mon[89904]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-03-22T05:49:31.212 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:30 smithi005 ceph-mon[89904]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:49:31.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:30 smithi186 ceph-mon[67903]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:49:31.294 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:49:31.294 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:49:31.294 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:49:31.295 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:49:31.295 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:49:31.295 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:49:31.295 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:49:31.295 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:49:31.295 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:49:31.295 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:49:31.295 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:49:31.295 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:49:31.295 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:49:31.295 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:49:31.295 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:49:31.295 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:49:31.295 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:49:31.295 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:49:31.296 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:49:31.296 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:49:31.296 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:49:31.296 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:49:31.296 INFO:journalctl@ceph.mon.b.smithi186.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:49:31.296 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:49:31.296 INFO:journalctl@ceph.mon.b.smithi186.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:49:31.296 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:49:31.296 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:49:31.296 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:49:31.296 INFO:journalctl@ceph.mon.b.smithi186.stdout: return future.result(timeout) 2024-03-22T05:49:31.296 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:49:31.296 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.__get_result() 2024-03-22T05:49:31.296 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:49:31.296 INFO:journalctl@ceph.mon.b.smithi186.stdout: raise self._exception 2024-03-22T05:49:31.296 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:49:31.297 INFO:journalctl@ceph.mon.b.smithi186.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:49:31.297 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:49:31.297 INFO:journalctl@ceph.mon.b.smithi186.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:49:31.297 INFO:journalctl@ceph.mon.b.smithi186.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:49:31.297 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:49:31.297 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:49:31.297 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:49:31.297 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:49:31.297 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:49:31.297 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:49:31.297 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:49:31.297 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:49:31.297 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:49:31.297 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:49:31.297 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:49:31.298 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:49:31.298 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:49:31.298 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:49:31.298 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:49:31.298 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:49:31.298 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:49:31.298 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:49:31.298 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:49:31.298 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:49:31.298 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:30 smithi186 ceph-mon[67903]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-22T05:49:31.298 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:30 smithi186 ceph-mon[67903]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-03-22T05:49:31.298 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:30 smithi186 ceph-mon[67903]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail 2024-03-22T05:49:33.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:32 smithi005 ceph-mon[84705]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 390 B/s rd, 0 op/s 2024-03-22T05:49:33.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:32 smithi005 ceph-mon[89904]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 390 B/s rd, 0 op/s 2024-03-22T05:49:33.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:32 smithi186 ceph-mon[67903]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 390 B/s rd, 0 op/s 2024-03-22T05:49:35.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:34 smithi005 ceph-mon[84705]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:49:35.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:34 smithi005 ceph-mon[84705]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 311 B/s rd, 0 op/s 2024-03-22T05:49:35.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:34 smithi005 ceph-mon[89904]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:49:35.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:34 smithi005 ceph-mon[89904]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 311 B/s rd, 0 op/s 2024-03-22T05:49:35.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:34 smithi186 ceph-mon[67903]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:49:35.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:34 smithi186 ceph-mon[67903]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 311 B/s rd, 0 op/s 2024-03-22T05:49:36.791 INFO:teuthology.orchestra.run.smithi005.stdout:true 2024-03-22T05:49:37.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:36 smithi005 ceph-mon[89904]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 311 B/s rd, 0 op/s 2024-03-22T05:49:37.202 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:36 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:49:36] "GET /metrics HTTP/1.1" 200 38094 "" "Prometheus/2.43.0" 2024-03-22T05:49:37.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:36 smithi005 ceph-mon[84705]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 311 B/s rd, 0 op/s 2024-03-22T05:49:37.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:36 smithi186 ceph-mon[67903]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 311 B/s rd, 0 op/s 2024-03-22T05:49:37.465 INFO:teuthology.orchestra.run.smithi005.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T05:49:37.465 INFO:teuthology.orchestra.run.smithi005.stdout:alertmanager.a smithi005 *:9093,9094 running (7m) 14s ago 29m 28.0M - 0.25.0 c8568f914cd2 ee99dfc435e8 2024-03-22T05:49:37.465 INFO:teuthology.orchestra.run.smithi005.stdout:grafana.a smithi186 *:3000 running (6m) 4m ago 28m 81.5M - 9.4.12 2bacad6d85d8 16fd57dbbd71 2024-03-22T05:49:37.465 INFO:teuthology.orchestra.run.smithi005.stdout:iscsi.foo.smithi005.mcdwbp smithi005 unknown 14s ago 28m 64.6M - 3.5 e1d6a67b021e 5215e03bf81b 2024-03-22T05:49:37.465 INFO:teuthology.orchestra.run.smithi005.stdout:mgr.x smithi186 *:8443,9283,8765 running (8m) 4m ago 33m 448M - 19.0.0-2299-gea01217f aa461e7c5e7f c556421a73f3 2024-03-22T05:49:37.465 INFO:teuthology.orchestra.run.smithi005.stdout:mgr.y smithi005 *:8443,9283,8765 running (18m) 14s ago 35m 511M - 19.0.0-2299-gea01217f aa461e7c5e7f 62a813eebc0d 2024-03-22T05:49:37.466 INFO:teuthology.orchestra.run.smithi005.stdout:mon.a smithi005 running (4m) 14s ago 35m 55.3M 2048M 19.0.0-2299-gea01217f aa461e7c5e7f 111a2aa24dd9 2024-03-22T05:49:37.466 INFO:teuthology.orchestra.run.smithi005.stdout:mon.b smithi186 running (5m) 4m ago 33m 39.1M 2048M 19.0.0-2299-gea01217f aa461e7c5e7f c1798feac15b 2024-03-22T05:49:37.466 INFO:teuthology.orchestra.run.smithi005.stdout:mon.c smithi005 running (4m) 14s ago 33m 42.0M 2048M 19.0.0-2299-gea01217f aa461e7c5e7f 708c72ff74f5 2024-03-22T05:49:37.466 INFO:teuthology.orchestra.run.smithi005.stdout:node-exporter.a smithi005 *:9100 running (7m) 14s ago 29m 16.8M - 1.5.0 0da6a335fe13 f635cd72b9d3 2024-03-22T05:49:37.466 INFO:teuthology.orchestra.run.smithi005.stdout:node-exporter.b smithi186 *:9100 running (7m) 4m ago 29m 15.1M - 1.5.0 0da6a335fe13 d7e0b3d20191 2024-03-22T05:49:37.466 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0 smithi005 running (2m) 14s ago 33m 44.8M 2993M 19.0.0-2299-gea01217f aa461e7c5e7f 31b6cdee5f5b 2024-03-22T05:49:37.466 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1 smithi005 running (2m) 14s ago 32m 44.2M 2993M 19.0.0-2299-gea01217f aa461e7c5e7f 7d21bc9fd3c2 2024-03-22T05:49:37.466 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2 smithi005 running (77s) 14s ago 32m 42.0M 2993M 19.0.0-2299-gea01217f aa461e7c5e7f 628204cb7c98 2024-03-22T05:49:37.466 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3 smithi005 running (19s) 14s ago 32m 14.9M 2993M 19.0.0-2299-gea01217f aa461e7c5e7f d31a01a0230c 2024-03-22T05:49:37.466 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4 smithi186 running (31m) 4m ago 31m 56.4M 3248M 17.2.0 e1d6a67b021e cd963671f340 2024-03-22T05:49:37.466 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5 smithi186 running (31m) 4m ago 31m 51.7M 3248M 17.2.0 e1d6a67b021e f98c87f1a2cd 2024-03-22T05:49:37.466 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6 smithi186 running (30m) 4m ago 30m 52.0M 3248M 17.2.0 e1d6a67b021e 704ac49c68c4 2024-03-22T05:49:37.466 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7 smithi186 running (30m) 4m ago 30m 54.3M 3248M 17.2.0 e1d6a67b021e f415b90edf63 2024-03-22T05:49:37.466 INFO:teuthology.orchestra.run.smithi005.stdout:prometheus.a smithi186 *:9095 running (7m) 4m ago 29m 67.2M - 2.43.0 a07b618ecd1d a5a043588d6d 2024-03-22T05:49:37.467 INFO:teuthology.orchestra.run.smithi005.stdout:rgw.foo.smithi005.exdhqu smithi005 *:8000 running (28m) 14s ago 28m 81.5M - 17.2.0 e1d6a67b021e 6bb27fe11553 2024-03-22T05:49:37.467 INFO:teuthology.orchestra.run.smithi005.stdout:rgw.foo.smithi186.mcgodk smithi186 *:8000 running (28m) 4m ago 28m 78.2M - 17.2.0 e1d6a67b021e 19b5c20a34f7 2024-03-22T05:49:37.885 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-03-22T05:49:37.885 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": { 2024-03-22T05:49:37.885 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev)": 3 2024-03-22T05:49:37.886 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:49:37.886 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr": { 2024-03-22T05:49:37.886 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev)": 2 2024-03-22T05:49:37.886 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:49:37.886 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": { 2024-03-22T05:49:37.886 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2024-03-22T05:49:37.886 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev)": 4 2024-03-22T05:49:37.886 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:49:37.886 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw": { 2024-03-22T05:49:37.886 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-22T05:49:37.886 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:49:37.886 INFO:teuthology.orchestra.run.smithi005.stdout: "overall": { 2024-03-22T05:49:37.886 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-03-22T05:49:37.886 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev)": 9 2024-03-22T05:49:37.886 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-03-22T05:49:37.887 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-03-22T05:49:38.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:37 smithi005 ceph-mon[89904]: from='client.54474 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:49:38.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:37 smithi005 ceph-mon[89904]: from='client.64414 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:49:38.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:37 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:49:38.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:37 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:49:38.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:37 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:49:38.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:37 smithi005 ceph-mon[89904]: from='client.? 172.21.15.5:0/512745860' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:49:38.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:37 smithi005 ceph-mon[84705]: from='client.54474 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:49:38.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:37 smithi005 ceph-mon[84705]: from='client.64414 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:49:38.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:37 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:49:38.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:37 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:49:38.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:37 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:49:38.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:37 smithi005 ceph-mon[84705]: from='client.? 172.21.15.5:0/512745860' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:49:38.229 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-03-22T05:49:38.229 INFO:teuthology.orchestra.run.smithi005.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9", 2024-03-22T05:49:38.229 INFO:teuthology.orchestra.run.smithi005.stdout: "in_progress": true, 2024-03-22T05:49:38.229 INFO:teuthology.orchestra.run.smithi005.stdout: "which": "Upgrading daemons of type(s) crash,osd", 2024-03-22T05:49:38.229 INFO:teuthology.orchestra.run.smithi005.stdout: "services_complete": [], 2024-03-22T05:49:38.229 INFO:teuthology.orchestra.run.smithi005.stdout: "progress": "4/8 daemons upgraded", 2024-03-22T05:49:38.229 INFO:teuthology.orchestra.run.smithi005.stdout: "message": "Currently upgrading osd daemons", 2024-03-22T05:49:38.229 INFO:teuthology.orchestra.run.smithi005.stdout: "is_paused": false 2024-03-22T05:49:38.229 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-03-22T05:49:38.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:37 smithi186 ceph-mon[67903]: from='client.54474 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:49:38.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:37 smithi186 ceph-mon[67903]: from='client.64414 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:49:38.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:37 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:49:38.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:37 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:49:38.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:37 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:49:38.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:37 smithi186 ceph-mon[67903]: from='client.? 172.21.15.5:0/512745860' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:49:39.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:38 smithi186 ceph-mon[67903]: from='client.54486 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:49:39.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:38 smithi186 ceph-mon[67903]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:49:39.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:38 smithi186 ceph-mon[67903]: from='client.64432 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:49:39.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:38 smithi005 ceph-mon[84705]: from='client.54486 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:49:39.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:38 smithi005 ceph-mon[84705]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:49:39.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:38 smithi005 ceph-mon[84705]: from='client.64432 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:49:39.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:38 smithi005 ceph-mon[89904]: from='client.54486 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:49:39.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:38 smithi005 ceph-mon[89904]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:49:39.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:38 smithi005 ceph-mon[89904]: from='client.64432 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:49:41.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:40 smithi186 ceph-mon[67903]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:49:41.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:40 smithi005 ceph-mon[84705]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:49:41.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:40 smithi005 ceph-mon[89904]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:49:43.290 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:42 smithi005 ceph-mon[84705]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:49:43.290 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:42 smithi005 ceph-mon[89904]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:49:43.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:42 smithi186 ceph-mon[67903]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:49:45.250 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:44 smithi186 ceph-mon[67903]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:49:45.250 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:44 smithi186 ceph-mon[67903]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:49:45.250 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:44 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-22T05:49:45.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:44 smithi005 ceph-mon[84705]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:49:45.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:44 smithi005 ceph-mon[84705]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:49:45.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:44 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-22T05:49:45.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:44 smithi005 ceph-mon[89904]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:49:45.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:44 smithi005 ceph-mon[89904]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:49:45.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:44 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-22T05:49:46.227 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:45 smithi186 ceph-mon[67903]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-22T05:49:46.228 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:45 smithi186 ceph-mon[67903]: Upgrade: osd.4 is safe to restart 2024-03-22T05:49:46.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:45 smithi005 ceph-mon[84705]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-22T05:49:46.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:45 smithi005 ceph-mon[84705]: Upgrade: osd.4 is safe to restart 2024-03-22T05:49:46.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:45 smithi005 ceph-mon[89904]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-22T05:49:46.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:45 smithi005 ceph-mon[89904]: Upgrade: osd.4 is safe to restart 2024-03-22T05:49:47.285 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:46 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:49:46] "GET /metrics HTTP/1.1" 200 38249 "" "Prometheus/2.43.0" 2024-03-22T05:49:47.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:47 smithi186 ceph-mon[67903]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:49:47.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:47 smithi186 ceph-mon[67903]: Upgrade: Updating osd.4 2024-03-22T05:49:47.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:47 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:49:47.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:47 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-22T05:49:47.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:47 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:49:47.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:47 smithi186 ceph-mon[67903]: Deploying daemon osd.4 on smithi186 2024-03-22T05:49:47.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:47 smithi005 ceph-mon[84705]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:49:47.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:47 smithi005 ceph-mon[84705]: Upgrade: Updating osd.4 2024-03-22T05:49:47.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:47 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:49:47.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:47 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-22T05:49:47.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:47 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:49:47.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:47 smithi005 ceph-mon[84705]: Deploying daemon osd.4 on smithi186 2024-03-22T05:49:47.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:47 smithi005 ceph-mon[89904]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:49:47.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:47 smithi005 ceph-mon[89904]: Upgrade: Updating osd.4 2024-03-22T05:49:47.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:47 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:49:47.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:47 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-22T05:49:47.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:47 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:49:47.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:47 smithi005 ceph-mon[89904]: Deploying daemon osd.4 on smithi186 2024-03-22T05:49:49.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:49 smithi186 ceph-mon[67903]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:49:49.544 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 22 05:49:49 smithi186 systemd[1]: Stopping Ceph osd.4 for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:49:49.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:49 smithi005 ceph-mon[84705]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:49:49.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:49 smithi005 ceph-mon[89904]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:49:50.044 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 22 05:49:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-4[29127]: 2024-03-22T05:49:49.761+0000 7f10aeeeb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T05:49:50.044 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 22 05:49:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-4[29127]: 2024-03-22T05:49:49.762+0000 7f10aeeeb700 -1 osd.4 125 *** Got signal Terminated *** 2024-03-22T05:49:50.044 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 22 05:49:49 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-4[29127]: 2024-03-22T05:49:49.762+0000 7f10aeeeb700 -1 osd.4 125 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-22T05:49:50.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:50 smithi186 ceph-mon[67903]: osd.4 marked itself down and dead 2024-03-22T05:49:50.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:50 smithi005 ceph-mon[84705]: osd.4 marked itself down and dead 2024-03-22T05:49:50.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:50 smithi005 ceph-mon[89904]: osd.4 marked itself down and dead 2024-03-22T05:49:50.980 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 22 05:49:50 smithi186 podman[77609]: 2024-03-22 05:49:50.750965775 +0000 UTC m=+1.132604877 container died cd963671f340eb3575e388069a50cb34e062ce1020889f3b7d3ae322090fe277 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-4, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, version=8, ceph=True, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, release=754, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=, vcs-type=git, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD) 2024-03-22T05:49:51.263 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 22 05:49:51 smithi186 podman[77609]: 2024-03-22 05:49:51.071074718 +0000 UTC m=+1.452713819 container cleanup cd963671f340eb3575e388069a50cb34e062ce1020889f3b7d3ae322090fe277 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-4, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, architecture=x86_64, vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, RELEASE=HEAD, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754) 2024-03-22T05:49:51.263 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 22 05:49:51 smithi186 bash[77609]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-4 2024-03-22T05:49:51.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:51 smithi186 ceph-mon[67903]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:49:51.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:51 smithi186 ceph-mon[67903]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-22T05:49:51.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:51 smithi186 ceph-mon[67903]: osdmap e126: 8 total, 7 up, 8 in 2024-03-22T05:49:51.544 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 22 05:49:51 smithi186 podman[77637]: 2024-03-22 05:49:51.263873554 +0000 UTC m=+0.509701648 container remove cd963671f340eb3575e388069a50cb34e062ce1020889f3b7d3ae322090fe277 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-4, maintainer=Guillaume Abrioux , name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vendor=Red Hat, Inc., version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, GIT_CLEAN=True, io.buildah.version=1.19.8, release=754, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public) 2024-03-22T05:49:51.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:51 smithi005 ceph-mon[84705]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:49:51.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:51 smithi005 ceph-mon[84705]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-22T05:49:51.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:51 smithi005 ceph-mon[84705]: osdmap e126: 8 total, 7 up, 8 in 2024-03-22T05:49:51.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:51 smithi005 ceph-mon[89904]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:49:51.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:51 smithi005 ceph-mon[89904]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-22T05:49:51.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:51 smithi005 ceph-mon[89904]: osdmap e126: 8 total, 7 up, 8 in 2024-03-22T05:49:52.025 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 22 05:49:51 smithi186 podman[77728]: 2024-03-22 05:49:51.603088599 +0000 UTC m=+0.019265159 image pull aa461e7c5e7f322b169c789b389d594b994ea7ffcbc9e24e563a3ec9b69bc3c1 quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9 2024-03-22T05:49:52.025 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 22 05:49:51 smithi186 podman[77728]: 2024-03-22 05:49:51.723203937 +0000 UTC m=+0.139380536 container create 57e6003f7f5ee1f66f0037a89a2c3ed7f3d435992c128bb0f778067c150c0f48 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-4-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-22T05:49:52.025 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 22 05:49:51 smithi186 podman[77728]: 2024-03-22 05:49:51.916937801 +0000 UTC m=+0.333114363 container init 57e6003f7f5ee1f66f0037a89a2c3ed7f3d435992c128bb0f778067c150c0f48 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-4-deactivate, org.label-schema.vendor=CentOS, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-22T05:49:52.025 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 22 05:49:51 smithi186 podman[77728]: 2024-03-22 05:49:51.920059552 +0000 UTC m=+0.336236113 container start 57e6003f7f5ee1f66f0037a89a2c3ed7f3d435992c128bb0f778067c150c0f48 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-4-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-22T05:49:52.025 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 22 05:49:52 smithi186 podman[77728]: 2024-03-22 05:49:52.02559714 +0000 UTC m=+0.441773717 container attach 57e6003f7f5ee1f66f0037a89a2c3ed7f3d435992c128bb0f778067c150c0f48 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-4-deactivate, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-22T05:49:52.294 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 22 05:49:52 smithi186 podman[77728]: 2024-03-22 05:49:52.209222214 +0000 UTC m=+0.625398765 container died 57e6003f7f5ee1f66f0037a89a2c3ed7f3d435992c128bb0f778067c150c0f48 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-4-deactivate, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217) 2024-03-22T05:49:52.561 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:52 smithi186 ceph-mon[67903]: osdmap e127: 8 total, 7 up, 8 in 2024-03-22T05:49:52.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:52 smithi005 ceph-mon[84705]: osdmap e127: 8 total, 7 up, 8 in 2024-03-22T05:49:52.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:52 smithi005 ceph-mon[89904]: osdmap e127: 8 total, 7 up, 8 in 2024-03-22T05:49:53.265 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 22 05:49:52 smithi186 podman[77754]: 2024-03-22 05:49:52.829165295 +0000 UTC m=+0.614448586 container remove 57e6003f7f5ee1f66f0037a89a2c3ed7f3d435992c128bb0f778067c150c0f48 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-4-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, org.label-schema.license=GPLv2) 2024-03-22T05:49:53.265 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 22 05:49:52 smithi186 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@osd.4.service: Deactivated successfully. 2024-03-22T05:49:53.265 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 22 05:49:52 smithi186 systemd[1]: Stopped Ceph osd.4 for f59f36e2-e80a-11ee-95cd-87774f69a715. 2024-03-22T05:49:53.265 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 22 05:49:52 smithi186 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@osd.4.service: Consumed 17.480s CPU time. 2024-03-22T05:49:53.265 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 22 05:49:53 smithi186 systemd[1]: Starting Ceph osd.4 for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:49:53.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:53 smithi186 ceph-mon[67903]: pgmap v196: 161 pgs: 8 active+undersized+degraded, 15 stale+active+clean, 11 active+undersized, 127 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s; 35/723 objects degraded (4.841%) 2024-03-22T05:49:53.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:53 smithi186 ceph-mon[67903]: Health check failed: Degraded data redundancy: 35/723 objects degraded (4.841%), 8 pgs degraded (PG_DEGRADED) 2024-03-22T05:49:53.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:53 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:49:53.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:53 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:49:53.544 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 22 05:49:53 smithi186 podman[77861]: 2024-03-22 05:49:53.168256841 +0000 UTC m=+0.019045868 image pull aa461e7c5e7f322b169c789b389d594b994ea7ffcbc9e24e563a3ec9b69bc3c1 quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9 2024-03-22T05:49:53.544 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 22 05:49:53 smithi186 podman[77861]: 2024-03-22 05:49:53.288590019 +0000 UTC m=+0.139379099 container create 7e20f9f258214e6b97687f9ec6cf84e09b1c4592035a7600874b2d45d0c8ed7b (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-4-activate, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_CLEAN=True) 2024-03-22T05:49:53.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:53 smithi005 ceph-mon[84705]: pgmap v196: 161 pgs: 8 active+undersized+degraded, 15 stale+active+clean, 11 active+undersized, 127 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s; 35/723 objects degraded (4.841%) 2024-03-22T05:49:53.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:53 smithi005 ceph-mon[84705]: Health check failed: Degraded data redundancy: 35/723 objects degraded (4.841%), 8 pgs degraded (PG_DEGRADED) 2024-03-22T05:49:53.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:53 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:49:53.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:53 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:49:53.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:53 smithi005 ceph-mon[89904]: pgmap v196: 161 pgs: 8 active+undersized+degraded, 15 stale+active+clean, 11 active+undersized, 127 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s; 35/723 objects degraded (4.841%) 2024-03-22T05:49:53.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:53 smithi005 ceph-mon[89904]: Health check failed: Degraded data redundancy: 35/723 objects degraded (4.841%), 8 pgs degraded (PG_DEGRADED) 2024-03-22T05:49:53.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:53 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:49:53.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:53 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:49:53.997 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 22 05:49:53 smithi186 podman[77861]: 2024-03-22 05:49:53.549728216 +0000 UTC m=+0.400517238 container init 7e20f9f258214e6b97687f9ec6cf84e09b1c4592035a7600874b2d45d0c8ed7b (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-4-activate, org.label-schema.schema-version=1.0, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-03-22T05:49:53.997 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 22 05:49:53 smithi186 podman[77861]: 2024-03-22 05:49:53.553817288 +0000 UTC m=+0.404606314 container start 7e20f9f258214e6b97687f9ec6cf84e09b1c4592035a7600874b2d45d0c8ed7b (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-4-activate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, io.buildah.version=1.33.5) 2024-03-22T05:49:53.997 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 22 05:49:53 smithi186 podman[77861]: 2024-03-22 05:49:53.641212932 +0000 UTC m=+0.492001959 container attach 7e20f9f258214e6b97687f9ec6cf84e09b1c4592035a7600874b2d45d0c8ed7b (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-4-activate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240319, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2) 2024-03-22T05:49:54.294 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 22 05:49:53 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-4-activate[77876]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-22T05:49:54.294 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 22 05:49:53 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-4-activate[77876]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-22T05:49:54.295 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 22 05:49:53 smithi186 bash[77861]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-22T05:49:54.295 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 22 05:49:53 smithi186 bash[77861]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-22T05:49:54.295 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 22 05:49:53 smithi186 bash[77861]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-22T05:49:54.295 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 22 05:49:53 smithi186 bash[77861]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-22T05:49:54.295 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 22 05:49:53 smithi186 bash[77861]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-03-22T05:49:54.295 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 22 05:49:53 smithi186 bash[77861]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-22T05:49:54.296 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 22 05:49:53 smithi186 bash[77861]: --> ceph-volume raw activate successful for osd ID: 4 2024-03-22T05:49:54.296 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 22 05:49:53 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-4-activate[77876]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-22T05:49:54.296 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 22 05:49:53 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-4-activate[77876]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-22T05:49:54.296 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 22 05:49:53 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-4-activate[77876]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-03-22T05:49:54.296 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 22 05:49:53 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-4-activate[77876]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-22T05:49:54.296 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 22 05:49:53 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-4-activate[77876]: --> ceph-volume raw activate successful for osd ID: 4 2024-03-22T05:49:54.296 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 22 05:49:54 smithi186 podman[77861]: 2024-03-22 05:49:54.022315844 +0000 UTC m=+0.873104855 container died 7e20f9f258214e6b97687f9ec6cf84e09b1c4592035a7600874b2d45d0c8ed7b (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-4-activate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-22T05:49:54.911 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 22 05:49:54 smithi186 podman[77997]: 2024-03-22 05:49:54.586535169 +0000 UTC m=+0.558086667 container remove 7e20f9f258214e6b97687f9ec6cf84e09b1c4592035a7600874b2d45d0c8ed7b (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-4-activate, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, CEPH_POINT_RELEASE=) 2024-03-22T05:49:54.911 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 22 05:49:54 smithi186 podman[78061]: 2024-03-22 05:49:54.706120232 +0000 UTC m=+0.017894309 image pull aa461e7c5e7f322b169c789b389d594b994ea7ffcbc9e24e563a3ec9b69bc3c1 quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9 2024-03-22T05:49:54.912 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 22 05:49:54 smithi186 podman[78061]: 2024-03-22 05:49:54.8125187 +0000 UTC m=+0.124292778 container create 8f7619b726f372801ef2cded85833a5a056a0c8a7e3fcc9474923d4bb0fc99ce (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-4, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-22T05:49:55.294 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 22 05:49:55 smithi186 podman[78061]: 2024-03-22 05:49:55.007831503 +0000 UTC m=+0.319605582 container init 8f7619b726f372801ef2cded85833a5a056a0c8a7e3fcc9474923d4bb0fc99ce (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-4, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, org.label-schema.build-date=20240319, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-22T05:49:55.294 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 22 05:49:55 smithi186 podman[78061]: 2024-03-22 05:49:55.010807753 +0000 UTC m=+0.322581840 container start 8f7619b726f372801ef2cded85833a5a056a0c8a7e3fcc9474923d4bb0fc99ce (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-4, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-22T05:49:55.294 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 22 05:49:55 smithi186 ceph-osd[78080]: -- 172.21.15.186:0/1757355712 <== mon.2 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x564bfe77f680 con 0x564bfe749400 2024-03-22T05:49:55.294 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 22 05:49:55 smithi186 bash[78061]: 8f7619b726f372801ef2cded85833a5a056a0c8a7e3fcc9474923d4bb0fc99ce 2024-03-22T05:49:55.294 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 22 05:49:55 smithi186 systemd[1]: Started Ceph osd.4 for f59f36e2-e80a-11ee-95cd-87774f69a715. 2024-03-22T05:49:55.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:55 smithi005 ceph-mon[84705]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:49:55.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:55 smithi005 ceph-mon[84705]: pgmap v197: 161 pgs: 22 active+undersized+degraded, 3 stale+active+clean, 36 active+undersized, 100 active+clean; 585 KiB data, 174 MiB used, 715 GiB / 715 GiB avail; 106/723 objects degraded (14.661%) 2024-03-22T05:49:55.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:55 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:49:55.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:55 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:49:55.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:55 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:49:55.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:55 smithi005 ceph-mon[89904]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:49:55.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:55 smithi005 ceph-mon[89904]: pgmap v197: 161 pgs: 22 active+undersized+degraded, 3 stale+active+clean, 36 active+undersized, 100 active+clean; 585 KiB data, 174 MiB used, 715 GiB / 715 GiB avail; 106/723 objects degraded (14.661%) 2024-03-22T05:49:55.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:55 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:49:55.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:55 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:49:55.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:55 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:49:55.794 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 22 05:49:55 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-4[78076]: 2024-03-22T05:49:55.547+0000 7f91d67077c0 -1 Falling back to public interface 2024-03-22T05:49:55.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:55 smithi186 ceph-mon[67903]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:49:55.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:55 smithi186 ceph-mon[67903]: pgmap v197: 161 pgs: 22 active+undersized+degraded, 3 stale+active+clean, 36 active+undersized, 100 active+clean; 585 KiB data, 174 MiB used, 715 GiB / 715 GiB avail; 106/723 objects degraded (14.661%) 2024-03-22T05:49:55.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:55 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:49:55.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:55 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:49:55.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:55 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:49:57.310 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:49:56 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:49:56] "GET /metrics HTTP/1.1" 200 38245 "" "Prometheus/2.43.0" 2024-03-22T05:49:57.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:57 smithi005 ceph-mon[84705]: pgmap v198: 161 pgs: 26 active+undersized+degraded, 43 active+undersized, 92 active+clean; 585 KiB data, 174 MiB used, 715 GiB / 715 GiB avail; 125/723 objects degraded (17.289%) 2024-03-22T05:49:57.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:57 smithi005 ceph-mon[89904]: pgmap v198: 161 pgs: 26 active+undersized+degraded, 43 active+undersized, 92 active+clean; 585 KiB data, 174 MiB used, 715 GiB / 715 GiB avail; 125/723 objects degraded (17.289%) 2024-03-22T05:49:57.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:57 smithi186 ceph-mon[67903]: pgmap v198: 161 pgs: 26 active+undersized+degraded, 43 active+undersized, 92 active+clean; 585 KiB data, 174 MiB used, 715 GiB / 715 GiB avail; 125/723 objects degraded (17.289%) 2024-03-22T05:49:58.511 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:58 smithi186 ceph-mon[67903]: from='osd.4 [v2:172.21.15.186:6800/3672682497,v1:172.21.15.186:6801/3672682497]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-22T05:49:58.512 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:58 smithi186 ceph-mon[67903]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-22T05:49:58.512 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 22 05:49:58 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-4[78076]: 2024-03-22T05:49:58.252+0000 7f91d67077c0 -1 osd.4 0 read_superblock omap replica is missing. 2024-03-22T05:49:58.512 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 22 05:49:58 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-4[78076]: 2024-03-22T05:49:58.284+0000 7f91d67077c0 -1 osd.4 125 log_to_monitors true 2024-03-22T05:49:58.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:58 smithi005 ceph-mon[84705]: from='osd.4 [v2:172.21.15.186:6800/3672682497,v1:172.21.15.186:6801/3672682497]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-22T05:49:58.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:58 smithi005 ceph-mon[84705]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-22T05:49:58.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:58 smithi005 ceph-mon[89904]: from='osd.4 [v2:172.21.15.186:6800/3672682497,v1:172.21.15.186:6801/3672682497]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-22T05:49:58.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:58 smithi005 ceph-mon[89904]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-22T05:49:59.063 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 22 05:49:58 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-4[78076]: 2024-03-22T05:49:58.801+0000 7f91c3b57700 -1 osd.4 125 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-22T05:49:59.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:59 smithi005 ceph-mon[89904]: pgmap v199: 161 pgs: 26 active+undersized+degraded, 43 active+undersized, 92 active+clean; 585 KiB data, 174 MiB used, 715 GiB / 715 GiB avail; 125/723 objects degraded (17.289%) 2024-03-22T05:49:59.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:59 smithi005 ceph-mon[89904]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-22T05:49:59.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:59 smithi005 ceph-mon[89904]: osdmap e128: 8 total, 7 up, 8 in 2024-03-22T05:49:59.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:59 smithi005 ceph-mon[89904]: from='osd.4 [v2:172.21.15.186:6800/3672682497,v1:172.21.15.186:6801/3672682497]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-03-22T05:49:59.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:59 smithi005 ceph-mon[89904]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-03-22T05:49:59.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:49:59 smithi005 ceph-mon[89904]: from='osd.4 ' entity='osd.4' 2024-03-22T05:49:59.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:59 smithi005 ceph-mon[84705]: pgmap v199: 161 pgs: 26 active+undersized+degraded, 43 active+undersized, 92 active+clean; 585 KiB data, 174 MiB used, 715 GiB / 715 GiB avail; 125/723 objects degraded (17.289%) 2024-03-22T05:49:59.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:59 smithi005 ceph-mon[84705]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-22T05:49:59.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:59 smithi005 ceph-mon[84705]: osdmap e128: 8 total, 7 up, 8 in 2024-03-22T05:49:59.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:59 smithi005 ceph-mon[84705]: from='osd.4 [v2:172.21.15.186:6800/3672682497,v1:172.21.15.186:6801/3672682497]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-03-22T05:49:59.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:59 smithi005 ceph-mon[84705]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-03-22T05:49:59.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:49:59 smithi005 ceph-mon[84705]: from='osd.4 ' entity='osd.4' 2024-03-22T05:49:59.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:59 smithi186 ceph-mon[67903]: pgmap v199: 161 pgs: 26 active+undersized+degraded, 43 active+undersized, 92 active+clean; 585 KiB data, 174 MiB used, 715 GiB / 715 GiB avail; 125/723 objects degraded (17.289%) 2024-03-22T05:49:59.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:59 smithi186 ceph-mon[67903]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-22T05:49:59.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:59 smithi186 ceph-mon[67903]: osdmap e128: 8 total, 7 up, 8 in 2024-03-22T05:49:59.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:59 smithi186 ceph-mon[67903]: from='osd.4 [v2:172.21.15.186:6800/3672682497,v1:172.21.15.186:6801/3672682497]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-03-22T05:49:59.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:59 smithi186 ceph-mon[67903]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-03-22T05:49:59.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:49:59 smithi186 ceph-mon[67903]: from='osd.4 ' entity='osd.4' 2024-03-22T05:50:00.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:00 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:50:00.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:00 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:50:00.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:00 smithi005 ceph-mon[89904]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-22T05:50:00.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:00 smithi005 ceph-mon[89904]: pgmap v202: 161 pgs: 26 active+undersized+degraded, 43 active+undersized, 92 active+clean; 585 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s; 125/723 objects degraded (17.289%) 2024-03-22T05:50:00.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:00 smithi005 ceph-mon[89904]: osd.4 [v2:172.21.15.186:6800/3672682497,v1:172.21.15.186:6801/3672682497] boot 2024-03-22T05:50:00.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:00 smithi005 ceph-mon[89904]: osdmap e129: 8 total, 8 up, 8 in 2024-03-22T05:50:00.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:00 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T05:50:00.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:00 smithi005 ceph-mon[89904]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s); Degraded data redundancy: 125/723 objects degraded (17.289%), 26 pgs degraded 2024-03-22T05:50:00.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:00 smithi005 ceph-mon[89904]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-22T05:50:00.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:00 smithi005 ceph-mon[89904]: daemon iscsi.foo.smithi005.mcdwbp on smithi005 is in unknown state 2024-03-22T05:50:00.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:00 smithi005 ceph-mon[89904]: [WRN] PG_DEGRADED: Degraded data redundancy: 125/723 objects degraded (17.289%), 26 pgs degraded 2024-03-22T05:50:00.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:00 smithi005 ceph-mon[89904]: pg 2.6 is active+undersized+degraded, acting [1,6] 2024-03-22T05:50:00.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:00 smithi005 ceph-mon[89904]: pg 2.b is active+undersized+degraded, acting [7,5] 2024-03-22T05:50:00.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:00 smithi005 ceph-mon[89904]: pg 2.f is active+undersized+degraded, acting [0,7] 2024-03-22T05:50:00.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:00 smithi005 ceph-mon[89904]: pg 2.1f is active+undersized+degraded, acting [0,7] 2024-03-22T05:50:00.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:00 smithi005 ceph-mon[89904]: pg 3.1 is active+undersized+degraded, acting [0,3] 2024-03-22T05:50:00.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:00 smithi005 ceph-mon[89904]: pg 3.3 is active+undersized+degraded, acting [0,6] 2024-03-22T05:50:00.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:00 smithi005 ceph-mon[89904]: pg 3.6 is active+undersized+degraded, acting [0,1] 2024-03-22T05:50:00.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:00 smithi005 ceph-mon[89904]: pg 3.9 is active+undersized+degraded, acting [2,7] 2024-03-22T05:50:00.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:00 smithi005 ceph-mon[89904]: pg 3.a is active+undersized+degraded, acting [6,1] 2024-03-22T05:50:00.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:00 smithi005 ceph-mon[89904]: pg 3.b is active+undersized+degraded, acting [3,0] 2024-03-22T05:50:00.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:00 smithi005 ceph-mon[89904]: pg 3.e is active+undersized+degraded, acting [7,1] 2024-03-22T05:50:00.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:00 smithi005 ceph-mon[89904]: pg 3.f is active+undersized+degraded, acting [7,0] 2024-03-22T05:50:00.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:00 smithi005 ceph-mon[89904]: pg 3.11 is active+undersized+degraded, acting [7,6] 2024-03-22T05:50:00.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:00 smithi005 ceph-mon[89904]: pg 3.13 is active+undersized+degraded, acting [7,2] 2024-03-22T05:50:00.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:00 smithi005 ceph-mon[89904]: pg 3.14 is active+undersized+degraded, acting [7,6] 2024-03-22T05:50:00.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:00 smithi005 ceph-mon[89904]: pg 3.15 is active+undersized+degraded, acting [7,3] 2024-03-22T05:50:00.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:00 smithi005 ceph-mon[89904]: pg 3.19 is active+undersized+degraded, acting [1,3] 2024-03-22T05:50:00.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:00 smithi005 ceph-mon[89904]: pg 3.1a is active+undersized+degraded, acting [1,2] 2024-03-22T05:50:00.955 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:00 smithi005 ceph-mon[89904]: pg 3.1b is active+undersized+degraded, acting [0,7] 2024-03-22T05:50:00.955 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:00 smithi005 ceph-mon[89904]: pg 3.1c is active+undersized+degraded, acting [5,1] 2024-03-22T05:50:00.955 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:00 smithi005 ceph-mon[89904]: pg 3.1d is active+undersized+degraded, acting [5,6] 2024-03-22T05:50:00.955 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:00 smithi005 ceph-mon[89904]: pg 4.9 is active+undersized+degraded, acting [1,3] 2024-03-22T05:50:00.955 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:00 smithi005 ceph-mon[89904]: pg 4.d is active+undersized+degraded, acting [2,1] 2024-03-22T05:50:00.955 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:00 smithi005 ceph-mon[89904]: pg 4.e is active+undersized+degraded, acting [6,1] 2024-03-22T05:50:00.955 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:00 smithi005 ceph-mon[89904]: pg 5.1c is active+undersized+degraded, acting [3,2] 2024-03-22T05:50:00.955 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:00 smithi005 ceph-mon[89904]: pg 6.1a is active+undersized+degraded, acting [5,1] 2024-03-22T05:50:00.955 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:00 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:50:00.955 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:00 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:50:00.955 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:00 smithi005 ceph-mon[84705]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-22T05:50:00.955 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:00 smithi005 ceph-mon[84705]: pgmap v202: 161 pgs: 26 active+undersized+degraded, 43 active+undersized, 92 active+clean; 585 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s; 125/723 objects degraded (17.289%) 2024-03-22T05:50:00.955 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:00 smithi005 ceph-mon[84705]: osd.4 [v2:172.21.15.186:6800/3672682497,v1:172.21.15.186:6801/3672682497] boot 2024-03-22T05:50:00.955 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:00 smithi005 ceph-mon[84705]: osdmap e129: 8 total, 8 up, 8 in 2024-03-22T05:50:00.956 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:00 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T05:50:00.956 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:00 smithi005 ceph-mon[84705]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s); Degraded data redundancy: 125/723 objects degraded (17.289%), 26 pgs degraded 2024-03-22T05:50:00.956 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:00 smithi005 ceph-mon[84705]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-22T05:50:00.956 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:00 smithi005 ceph-mon[84705]: daemon iscsi.foo.smithi005.mcdwbp on smithi005 is in unknown state 2024-03-22T05:50:00.956 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:00 smithi005 ceph-mon[84705]: [WRN] PG_DEGRADED: Degraded data redundancy: 125/723 objects degraded (17.289%), 26 pgs degraded 2024-03-22T05:50:00.956 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:00 smithi005 ceph-mon[84705]: pg 2.6 is active+undersized+degraded, acting [1,6] 2024-03-22T05:50:00.956 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:00 smithi005 ceph-mon[84705]: pg 2.b is active+undersized+degraded, acting [7,5] 2024-03-22T05:50:00.956 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:00 smithi005 ceph-mon[84705]: pg 2.f is active+undersized+degraded, acting [0,7] 2024-03-22T05:50:00.956 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:00 smithi005 ceph-mon[84705]: pg 2.1f is active+undersized+degraded, acting [0,7] 2024-03-22T05:50:00.956 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:00 smithi005 ceph-mon[84705]: pg 3.1 is active+undersized+degraded, acting [0,3] 2024-03-22T05:50:00.956 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:00 smithi005 ceph-mon[84705]: pg 3.3 is active+undersized+degraded, acting [0,6] 2024-03-22T05:50:00.956 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:00 smithi005 ceph-mon[84705]: pg 3.6 is active+undersized+degraded, acting [0,1] 2024-03-22T05:50:00.956 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:00 smithi005 ceph-mon[84705]: pg 3.9 is active+undersized+degraded, acting [2,7] 2024-03-22T05:50:00.956 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:00 smithi005 ceph-mon[84705]: pg 3.a is active+undersized+degraded, acting [6,1] 2024-03-22T05:50:00.956 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:00 smithi005 ceph-mon[84705]: pg 3.b is active+undersized+degraded, acting [3,0] 2024-03-22T05:50:00.956 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:00 smithi005 ceph-mon[84705]: pg 3.e is active+undersized+degraded, acting [7,1] 2024-03-22T05:50:00.957 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:00 smithi005 ceph-mon[84705]: pg 3.f is active+undersized+degraded, acting [7,0] 2024-03-22T05:50:00.957 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:00 smithi005 ceph-mon[84705]: pg 3.11 is active+undersized+degraded, acting [7,6] 2024-03-22T05:50:00.957 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:00 smithi005 ceph-mon[84705]: pg 3.13 is active+undersized+degraded, acting [7,2] 2024-03-22T05:50:00.957 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:00 smithi005 ceph-mon[84705]: pg 3.14 is active+undersized+degraded, acting [7,6] 2024-03-22T05:50:00.957 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:00 smithi005 ceph-mon[84705]: pg 3.15 is active+undersized+degraded, acting [7,3] 2024-03-22T05:50:00.957 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:00 smithi005 ceph-mon[84705]: pg 3.19 is active+undersized+degraded, acting [1,3] 2024-03-22T05:50:00.957 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:00 smithi005 ceph-mon[84705]: pg 3.1a is active+undersized+degraded, acting [1,2] 2024-03-22T05:50:00.957 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:00 smithi005 ceph-mon[84705]: pg 3.1b is active+undersized+degraded, acting [0,7] 2024-03-22T05:50:00.957 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:00 smithi005 ceph-mon[84705]: pg 3.1c is active+undersized+degraded, acting [5,1] 2024-03-22T05:50:00.957 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:00 smithi005 ceph-mon[84705]: pg 3.1d is active+undersized+degraded, acting [5,6] 2024-03-22T05:50:00.957 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:00 smithi005 ceph-mon[84705]: pg 4.9 is active+undersized+degraded, acting [1,3] 2024-03-22T05:50:00.957 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:00 smithi005 ceph-mon[84705]: pg 4.d is active+undersized+degraded, acting [2,1] 2024-03-22T05:50:00.957 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:00 smithi005 ceph-mon[84705]: pg 4.e is active+undersized+degraded, acting [6,1] 2024-03-22T05:50:00.957 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:00 smithi005 ceph-mon[84705]: pg 5.1c is active+undersized+degraded, acting [3,2] 2024-03-22T05:50:00.957 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:00 smithi005 ceph-mon[84705]: pg 6.1a is active+undersized+degraded, acting [5,1] 2024-03-22T05:50:01.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:00 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:50:01.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:00 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:50:01.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:00 smithi186 ceph-mon[67903]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-22T05:50:01.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:00 smithi186 ceph-mon[67903]: pgmap v202: 161 pgs: 26 active+undersized+degraded, 43 active+undersized, 92 active+clean; 585 KiB data, 184 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s; 125/723 objects degraded (17.289%) 2024-03-22T05:50:01.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:00 smithi186 ceph-mon[67903]: osd.4 [v2:172.21.15.186:6800/3672682497,v1:172.21.15.186:6801/3672682497] boot 2024-03-22T05:50:01.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:00 smithi186 ceph-mon[67903]: osdmap e129: 8 total, 8 up, 8 in 2024-03-22T05:50:01.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:00 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-22T05:50:01.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:00 smithi186 ceph-mon[67903]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s); Degraded data redundancy: 125/723 objects degraded (17.289%), 26 pgs degraded 2024-03-22T05:50:01.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:00 smithi186 ceph-mon[67903]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-22T05:50:01.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:00 smithi186 ceph-mon[67903]: daemon iscsi.foo.smithi005.mcdwbp on smithi005 is in unknown state 2024-03-22T05:50:01.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:00 smithi186 ceph-mon[67903]: [WRN] PG_DEGRADED: Degraded data redundancy: 125/723 objects degraded (17.289%), 26 pgs degraded 2024-03-22T05:50:01.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:00 smithi186 ceph-mon[67903]: pg 2.6 is active+undersized+degraded, acting [1,6] 2024-03-22T05:50:01.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:00 smithi186 ceph-mon[67903]: pg 2.b is active+undersized+degraded, acting [7,5] 2024-03-22T05:50:01.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:00 smithi186 ceph-mon[67903]: pg 2.f is active+undersized+degraded, acting [0,7] 2024-03-22T05:50:01.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:00 smithi186 ceph-mon[67903]: pg 2.1f is active+undersized+degraded, acting [0,7] 2024-03-22T05:50:01.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:00 smithi186 ceph-mon[67903]: pg 3.1 is active+undersized+degraded, acting [0,3] 2024-03-22T05:50:01.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:00 smithi186 ceph-mon[67903]: pg 3.3 is active+undersized+degraded, acting [0,6] 2024-03-22T05:50:01.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:00 smithi186 ceph-mon[67903]: pg 3.6 is active+undersized+degraded, acting [0,1] 2024-03-22T05:50:01.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:00 smithi186 ceph-mon[67903]: pg 3.9 is active+undersized+degraded, acting [2,7] 2024-03-22T05:50:01.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:00 smithi186 ceph-mon[67903]: pg 3.a is active+undersized+degraded, acting [6,1] 2024-03-22T05:50:01.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:00 smithi186 ceph-mon[67903]: pg 3.b is active+undersized+degraded, acting [3,0] 2024-03-22T05:50:01.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:00 smithi186 ceph-mon[67903]: pg 3.e is active+undersized+degraded, acting [7,1] 2024-03-22T05:50:01.046 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:00 smithi186 ceph-mon[67903]: pg 3.f is active+undersized+degraded, acting [7,0] 2024-03-22T05:50:01.046 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:00 smithi186 ceph-mon[67903]: pg 3.11 is active+undersized+degraded, acting [7,6] 2024-03-22T05:50:01.046 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:00 smithi186 ceph-mon[67903]: pg 3.13 is active+undersized+degraded, acting [7,2] 2024-03-22T05:50:01.046 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:00 smithi186 ceph-mon[67903]: pg 3.14 is active+undersized+degraded, acting [7,6] 2024-03-22T05:50:01.046 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:00 smithi186 ceph-mon[67903]: pg 3.15 is active+undersized+degraded, acting [7,3] 2024-03-22T05:50:01.046 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:00 smithi186 ceph-mon[67903]: pg 3.19 is active+undersized+degraded, acting [1,3] 2024-03-22T05:50:01.046 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:00 smithi186 ceph-mon[67903]: pg 3.1a is active+undersized+degraded, acting [1,2] 2024-03-22T05:50:01.046 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:00 smithi186 ceph-mon[67903]: pg 3.1b is active+undersized+degraded, acting [0,7] 2024-03-22T05:50:01.046 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:00 smithi186 ceph-mon[67903]: pg 3.1c is active+undersized+degraded, acting [5,1] 2024-03-22T05:50:01.046 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:00 smithi186 ceph-mon[67903]: pg 3.1d is active+undersized+degraded, acting [5,6] 2024-03-22T05:50:01.046 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:00 smithi186 ceph-mon[67903]: pg 4.9 is active+undersized+degraded, acting [1,3] 2024-03-22T05:50:01.046 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:00 smithi186 ceph-mon[67903]: pg 4.d is active+undersized+degraded, acting [2,1] 2024-03-22T05:50:01.046 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:00 smithi186 ceph-mon[67903]: pg 4.e is active+undersized+degraded, acting [6,1] 2024-03-22T05:50:01.046 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:00 smithi186 ceph-mon[67903]: pg 5.1c is active+undersized+degraded, acting [3,2] 2024-03-22T05:50:01.047 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:00 smithi186 ceph-mon[67903]: pg 6.1a is active+undersized+degraded, acting [5,1] 2024-03-22T05:50:02.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:01 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:50:02.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:01 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:50:02.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:01 smithi186 ceph-mon[67903]: osdmap e130: 8 total, 8 up, 8 in 2024-03-22T05:50:02.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:01 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:50:02.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:01 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:50:02.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:01 smithi005 ceph-mon[89904]: osdmap e130: 8 total, 8 up, 8 in 2024-03-22T05:50:02.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:01 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:50:02.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:01 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:50:02.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:01 smithi005 ceph-mon[84705]: osdmap e130: 8 total, 8 up, 8 in 2024-03-22T05:50:02.892 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:02 smithi186 ceph-mon[67903]: pgmap v204: 161 pgs: 21 active+undersized+degraded, 37 active+undersized, 103 active+clean; 585 KiB data, 185 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 102/723 objects degraded (14.108%) 2024-03-22T05:50:02.892 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:02 smithi186 ceph-mon[67903]: Health check update: Degraded data redundancy: 125/723 objects degraded (17.289%), 26 pgs degraded (PG_DEGRADED) 2024-03-22T05:50:03.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:02 smithi005 ceph-mon[84705]: pgmap v204: 161 pgs: 21 active+undersized+degraded, 37 active+undersized, 103 active+clean; 585 KiB data, 185 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 102/723 objects degraded (14.108%) 2024-03-22T05:50:03.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:02 smithi005 ceph-mon[84705]: Health check update: Degraded data redundancy: 125/723 objects degraded (17.289%), 26 pgs degraded (PG_DEGRADED) 2024-03-22T05:50:03.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:02 smithi005 ceph-mon[89904]: pgmap v204: 161 pgs: 21 active+undersized+degraded, 37 active+undersized, 103 active+clean; 585 KiB data, 185 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 102/723 objects degraded (14.108%) 2024-03-22T05:50:03.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:02 smithi005 ceph-mon[89904]: Health check update: Degraded data redundancy: 125/723 objects degraded (17.289%), 26 pgs degraded (PG_DEGRADED) 2024-03-22T05:50:05.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:05 smithi005 ceph-mon[84705]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:50:05.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:05 smithi005 ceph-mon[84705]: pgmap v205: 161 pgs: 2 active+undersized+degraded, 6 active+undersized, 153 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 12/723 objects degraded (1.660%) 2024-03-22T05:50:05.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:05 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:50:05.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:05 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:50:05.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:05 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:50:05.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:05 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:50:05.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:05 smithi005 ceph-mon[84705]: pgmap v206: 161 pgs: 2 active+undersized+degraded, 6 active+undersized, 153 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 883 B/s rd, 0 op/s; 12/723 objects degraded (1.660%) 2024-03-22T05:50:05.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:05 smithi005 ceph-mon[84705]: pgmap v207: 161 pgs: 2 active+undersized+degraded, 6 active+undersized, 153 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 12/723 objects degraded (1.660%) 2024-03-22T05:50:05.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:05 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:50:05.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:05 smithi005 ceph-mon[84705]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:50:05.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:05 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:50:05.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:05 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:50:05.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:05 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:50:05.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:05 smithi005 ceph-mon[84705]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:50:05.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:05 smithi005 ceph-mon[89904]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:50:05.204 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:05 smithi005 ceph-mon[89904]: pgmap v205: 161 pgs: 2 active+undersized+degraded, 6 active+undersized, 153 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 12/723 objects degraded (1.660%) 2024-03-22T05:50:05.204 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:05 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:50:05.204 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:05 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:50:05.204 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:05 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:50:05.204 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:05 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:50:05.204 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:05 smithi005 ceph-mon[89904]: pgmap v206: 161 pgs: 2 active+undersized+degraded, 6 active+undersized, 153 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 883 B/s rd, 0 op/s; 12/723 objects degraded (1.660%) 2024-03-22T05:50:05.204 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:05 smithi005 ceph-mon[89904]: pgmap v207: 161 pgs: 2 active+undersized+degraded, 6 active+undersized, 153 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 12/723 objects degraded (1.660%) 2024-03-22T05:50:05.204 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:05 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:50:05.204 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:05 smithi005 ceph-mon[89904]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:50:05.204 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:05 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:50:05.204 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:05 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:50:05.204 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:05 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:50:05.204 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:05 smithi005 ceph-mon[89904]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:50:05.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:05 smithi186 ceph-mon[67903]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:50:05.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:05 smithi186 ceph-mon[67903]: pgmap v205: 161 pgs: 2 active+undersized+degraded, 6 active+undersized, 153 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 12/723 objects degraded (1.660%) 2024-03-22T05:50:05.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:05 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:50:05.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:05 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:50:05.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:05 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:50:05.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:05 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:50:05.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:05 smithi186 ceph-mon[67903]: pgmap v206: 161 pgs: 2 active+undersized+degraded, 6 active+undersized, 153 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 883 B/s rd, 0 op/s; 12/723 objects degraded (1.660%) 2024-03-22T05:50:05.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:05 smithi186 ceph-mon[67903]: pgmap v207: 161 pgs: 2 active+undersized+degraded, 6 active+undersized, 153 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 12/723 objects degraded (1.660%) 2024-03-22T05:50:05.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:05 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:50:05.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:05 smithi186 ceph-mon[67903]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:50:05.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:05 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:50:05.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:05 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:50:05.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:05 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:50:05.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:05 smithi186 ceph-mon[67903]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:50:06.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:06 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:50:06.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:06 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:50:06.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:06 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:50:06.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:06 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:50:06.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:06 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-22T05:50:06.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:06 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:50:06.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:06 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:50:06.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:06 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:50:06.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:06 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:50:06.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:06 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-22T05:50:06.204 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:50:06.072+0000 7f7f0ecc9700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:50:06.204 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:50:06.204 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:50:06.204 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:50:06.204 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:50:06.204 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:50:06.204 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:50:06.204 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:50:06.204 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:50:06.204 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:50:06.205 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:50:06.205 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:50:06.205 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:50:06.205 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:50:06.205 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:50:06.205 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:50:06.205 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:50:06.205 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:50:06.205 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:50:06.205 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:50:06.205 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:50:06.205 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:50:06.205 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:50:06.205 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:50:06.205 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:50:06.206 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:50:06.206 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:50:06.206 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:50:06.206 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:50:06.206 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return future.result(timeout) 2024-03-22T05:50:06.206 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:50:06.206 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.__get_result() 2024-03-22T05:50:06.206 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:50:06.206 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: raise self._exception 2024-03-22T05:50:06.206 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:50:06.206 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: config_blobs=daemon_spec.final_config, 2024-03-22T05:50:06.206 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:50:06.206 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:50:06.206 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:50:06.207 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:50:06.207 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:50:06.207 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:50:06.207 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:50:06.207 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:50:06.207 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:50:06.207 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:50:06.207 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:50:06.207 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:50:06.207 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:50:06.207 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:50:06.207 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:50:06.207 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:50:06.208 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:50:06.208 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:50:06.208 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:50:06.208 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:50:06.208 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:50:06.208 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:50:06.208 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:50:06.208 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:50:06.086+0000 7f7f3e571700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-03-22T05:50:06.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:06 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:50:06.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:06 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:50:06.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:06 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:50:06.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:06 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:50:06.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:06 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-22T05:50:07.452 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:50:06] "GET /metrics HTTP/1.1" 200 38245 "" "Prometheus/2.43.0" 2024-03-22T05:50:07.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:07 smithi005 ceph-mon[89904]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:50:07.453 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:50:07.453 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:50:07.453 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:50:07.453 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:50:07.453 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:50:07.453 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:50:07.453 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:50:07.453 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:50:07.453 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:50:07.453 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:50:07.453 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:50:07.453 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:50:07.453 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:50:07.453 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:50:07.454 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:50:07.454 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:50:07.454 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:50:07.454 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:50:07.454 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:50:07.454 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:50:07.454 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:50:07.454 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:50:07.454 INFO:journalctl@ceph.mon.c.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:50:07.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:07 smithi005 ceph-mon[84705]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:50:07.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:50:07.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:50:07.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:50:07.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:50:07.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:50:07.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:50:07.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:50:07.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:50:07.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:50:07.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:50:07.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:50:07.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:50:07.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:50:07.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:50:07.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:50:07.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:50:07.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:50:07.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:50:07.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:50:07.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:50:07.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:50:07.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:50:07.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:50:07.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:50:07.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:50:07.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:50:07.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:50:07.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:50:07.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: return future.result(timeout) 2024-03-22T05:50:07.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:50:07.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.__get_result() 2024-03-22T05:50:07.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:50:07.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: raise self._exception 2024-03-22T05:50:07.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:50:07.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:50:07.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:50:07.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:50:07.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:50:07.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:50:07.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:50:07.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:50:07.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:50:07.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:50:07.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:50:07.458 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:50:07.458 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:50:07.458 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:50:07.458 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:50:07.458 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:50:07.458 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:50:07.458 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:50:07.458 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:50:07.458 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:50:07.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:50:07.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:50:07.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:50:07.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:50:07.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:50:07.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: return future.result(timeout) 2024-03-22T05:50:07.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:50:07.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.__get_result() 2024-03-22T05:50:07.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:50:07.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: raise self._exception 2024-03-22T05:50:07.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:50:07.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:50:07.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:50:07.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:50:07.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:50:07.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:50:07.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:50:07.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:50:07.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:50:07.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:50:07.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:50:07.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:50:07.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:50:07.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:50:07.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:50:07.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:50:07.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:50:07.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:50:07.461 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:50:07.461 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:50:07.461 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:50:07.461 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:50:07.461 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:50:07.461 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:50:07.461 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:50:07.461 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:07 smithi005 ceph-mon[89904]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-22T05:50:07.461 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:07 smithi005 ceph-mon[89904]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-03-22T05:50:07.461 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:07 smithi005 ceph-mon[89904]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 810 B/s rd, 0 op/s 2024-03-22T05:50:07.461 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:50:07.461 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:50:07.462 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:50:07.462 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:50:07.462 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:50:07.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:07 smithi005 ceph-mon[84705]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-22T05:50:07.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:07 smithi005 ceph-mon[84705]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-03-22T05:50:07.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:07 smithi005 ceph-mon[84705]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 810 B/s rd, 0 op/s 2024-03-22T05:50:07.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:07 smithi186 ceph-mon[67903]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:50:07.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:50:07.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:50:07.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:50:07.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:50:07.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:50:07.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:50:07.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:50:07.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:50:07.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:50:07.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:50:07.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:50:07.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:50:07.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:50:07.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:50:07.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:50:07.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:50:07.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:50:07.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:50:07.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:50:07.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:50:07.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:50:07.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:50:07.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:50:07.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:50:07.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:50:07.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:50:07.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:50:07.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:50:07.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: return future.result(timeout) 2024-03-22T05:50:07.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:50:07.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.__get_result() 2024-03-22T05:50:07.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:50:07.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: raise self._exception 2024-03-22T05:50:07.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:50:07.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:50:07.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:50:07.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:50:07.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:50:07.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:50:07.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:50:07.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:50:07.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:50:07.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:50:07.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:50:07.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:50:07.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:50:07.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:50:07.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:50:07.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:50:07.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:50:07.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:50:07.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:50:07.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:50:07.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:50:07.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:50:07.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:50:07.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:50:07.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:50:07.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:07 smithi186 ceph-mon[67903]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-22T05:50:07.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:07 smithi186 ceph-mon[67903]: Upgrade: unsafe to stop osd(s) at this time (2 PGs are or would become offline) 2024-03-22T05:50:07.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:07 smithi186 ceph-mon[67903]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 810 B/s rd, 0 op/s 2024-03-22T05:50:08.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:08 smithi005 ceph-mon[84705]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 12/723 objects degraded (1.660%), 2 pgs degraded) 2024-03-22T05:50:08.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:08 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:50:08.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:08 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:50:08.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:08 smithi005 ceph-mon[89904]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 12/723 objects degraded (1.660%), 2 pgs degraded) 2024-03-22T05:50:08.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:08 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:50:08.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:08 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:50:08.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:08 smithi186 ceph-mon[67903]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 12/723 objects degraded (1.660%), 2 pgs degraded) 2024-03-22T05:50:08.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:08 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:50:08.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:08 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:50:08.601 INFO:teuthology.orchestra.run.smithi005.stdout:true 2024-03-22T05:50:09.284 INFO:teuthology.orchestra.run.smithi005.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T05:50:09.284 INFO:teuthology.orchestra.run.smithi005.stdout:alertmanager.a smithi005 *:9093,9094 running (7m) 46s ago 29m 28.0M - 0.25.0 c8568f914cd2 ee99dfc435e8 2024-03-22T05:50:09.284 INFO:teuthology.orchestra.run.smithi005.stdout:grafana.a smithi186 *:3000 running (7m) 9s ago 29m 80.3M - 9.4.12 2bacad6d85d8 16fd57dbbd71 2024-03-22T05:50:09.284 INFO:teuthology.orchestra.run.smithi005.stdout:iscsi.foo.smithi005.mcdwbp smithi005 unknown 46s ago 28m 64.6M - 3.5 e1d6a67b021e 5215e03bf81b 2024-03-22T05:50:09.284 INFO:teuthology.orchestra.run.smithi005.stdout:mgr.x smithi186 *:8443,9283,8765 running (8m) 9s ago 34m 453M - 19.0.0-2299-gea01217f aa461e7c5e7f c556421a73f3 2024-03-22T05:50:09.284 INFO:teuthology.orchestra.run.smithi005.stdout:mgr.y smithi005 *:8443,9283,8765 running (18m) 46s ago 35m 511M - 19.0.0-2299-gea01217f aa461e7c5e7f 62a813eebc0d 2024-03-22T05:50:09.284 INFO:teuthology.orchestra.run.smithi005.stdout:mon.a smithi005 running (5m) 46s ago 35m 55.3M 2048M 19.0.0-2299-gea01217f aa461e7c5e7f 111a2aa24dd9 2024-03-22T05:50:09.284 INFO:teuthology.orchestra.run.smithi005.stdout:mon.b smithi186 running (6m) 9s ago 34m 50.1M 2048M 19.0.0-2299-gea01217f aa461e7c5e7f c1798feac15b 2024-03-22T05:50:09.285 INFO:teuthology.orchestra.run.smithi005.stdout:mon.c smithi005 running (4m) 46s ago 34m 42.0M 2048M 19.0.0-2299-gea01217f aa461e7c5e7f 708c72ff74f5 2024-03-22T05:50:09.285 INFO:teuthology.orchestra.run.smithi005.stdout:node-exporter.a smithi005 *:9100 running (8m) 46s ago 30m 16.8M - 1.5.0 0da6a335fe13 f635cd72b9d3 2024-03-22T05:50:09.285 INFO:teuthology.orchestra.run.smithi005.stdout:node-exporter.b smithi186 *:9100 running (8m) 9s ago 30m 15.0M - 1.5.0 0da6a335fe13 d7e0b3d20191 2024-03-22T05:50:09.285 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0 smithi005 running (3m) 46s ago 33m 44.8M 2993M 19.0.0-2299-gea01217f aa461e7c5e7f 31b6cdee5f5b 2024-03-22T05:50:09.285 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1 smithi005 running (2m) 46s ago 33m 44.2M 2993M 19.0.0-2299-gea01217f aa461e7c5e7f 7d21bc9fd3c2 2024-03-22T05:50:09.285 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2 smithi005 running (109s) 46s ago 32m 42.0M 2993M 19.0.0-2299-gea01217f aa461e7c5e7f 628204cb7c98 2024-03-22T05:50:09.285 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3 smithi005 running (51s) 46s ago 32m 14.9M 2993M 19.0.0-2299-gea01217f aa461e7c5e7f d31a01a0230c 2024-03-22T05:50:09.285 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4 smithi186 running (14s) 9s ago 32m 12.7M 3248M 19.0.0-2299-gea01217f aa461e7c5e7f 8f7619b726f3 2024-03-22T05:50:09.285 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5 smithi186 running (31m) 9s ago 31m 55.3M 3248M 17.2.0 e1d6a67b021e f98c87f1a2cd 2024-03-22T05:50:09.285 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6 smithi186 running (31m) 9s ago 31m 55.5M 3248M 17.2.0 e1d6a67b021e 704ac49c68c4 2024-03-22T05:50:09.285 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7 smithi186 running (31m) 9s ago 31m 58.7M 3248M 17.2.0 e1d6a67b021e f415b90edf63 2024-03-22T05:50:09.285 INFO:teuthology.orchestra.run.smithi005.stdout:prometheus.a smithi186 *:9095 running (8m) 9s ago 30m 69.9M - 2.43.0 a07b618ecd1d a5a043588d6d 2024-03-22T05:50:09.285 INFO:teuthology.orchestra.run.smithi005.stdout:rgw.foo.smithi005.exdhqu smithi005 *:8000 running (28m) 46s ago 28m 81.5M - 17.2.0 e1d6a67b021e 6bb27fe11553 2024-03-22T05:50:09.285 INFO:teuthology.orchestra.run.smithi005.stdout:rgw.foo.smithi186.mcgodk smithi186 *:8000 running (28m) 9s ago 28m 79.8M - 17.2.0 e1d6a67b021e 19b5c20a34f7 2024-03-22T05:50:09.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:09 smithi005 ceph-mon[84705]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-22T05:50:09.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:09 smithi005 ceph-mon[89904]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-22T05:50:09.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:09 smithi186 ceph-mon[67903]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-22T05:50:09.698 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-03-22T05:50:09.698 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": { 2024-03-22T05:50:09.698 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev)": 3 2024-03-22T05:50:09.698 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:50:09.698 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr": { 2024-03-22T05:50:09.699 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev)": 2 2024-03-22T05:50:09.699 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:50:09.699 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": { 2024-03-22T05:50:09.699 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3, 2024-03-22T05:50:09.699 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev)": 5 2024-03-22T05:50:09.699 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:50:09.699 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw": { 2024-03-22T05:50:09.699 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-22T05:50:09.699 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:50:09.699 INFO:teuthology.orchestra.run.smithi005.stdout: "overall": { 2024-03-22T05:50:09.699 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-03-22T05:50:09.699 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev)": 10 2024-03-22T05:50:09.699 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-03-22T05:50:09.699 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-03-22T05:50:10.041 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-03-22T05:50:10.041 INFO:teuthology.orchestra.run.smithi005.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9", 2024-03-22T05:50:10.041 INFO:teuthology.orchestra.run.smithi005.stdout: "in_progress": true, 2024-03-22T05:50:10.041 INFO:teuthology.orchestra.run.smithi005.stdout: "which": "Upgrading daemons of type(s) crash,osd", 2024-03-22T05:50:10.042 INFO:teuthology.orchestra.run.smithi005.stdout: "services_complete": [], 2024-03-22T05:50:10.042 INFO:teuthology.orchestra.run.smithi005.stdout: "progress": "5/8 daemons upgraded", 2024-03-22T05:50:10.042 INFO:teuthology.orchestra.run.smithi005.stdout: "message": "Currently upgrading osd daemons", 2024-03-22T05:50:10.042 INFO:teuthology.orchestra.run.smithi005.stdout: "is_paused": false 2024-03-22T05:50:10.042 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-03-22T05:50:10.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:10 smithi005 ceph-mon[84705]: from='client.54510 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:50:10.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:10 smithi005 ceph-mon[84705]: from='client.54516 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:50:10.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:10 smithi005 ceph-mon[84705]: from='client.64456 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:50:10.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:10 smithi005 ceph-mon[84705]: from='client.? 172.21.15.5:0/1971224750' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:50:10.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:10 smithi005 ceph-mon[89904]: from='client.54510 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:50:10.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:10 smithi005 ceph-mon[89904]: from='client.54516 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:50:10.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:10 smithi005 ceph-mon[89904]: from='client.64456 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:50:10.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:10 smithi005 ceph-mon[89904]: from='client.? 172.21.15.5:0/1971224750' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:50:10.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:10 smithi186 ceph-mon[67903]: from='client.54510 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:50:10.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:10 smithi186 ceph-mon[67903]: from='client.54516 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:50:10.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:10 smithi186 ceph-mon[67903]: from='client.64456 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:50:10.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:10 smithi186 ceph-mon[67903]: from='client.? 172.21.15.5:0/1971224750' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:50:11.420 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:11 smithi186 ceph-mon[67903]: from='client.64468 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:50:11.420 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:11 smithi186 ceph-mon[67903]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 615 B/s rd, 0 op/s 2024-03-22T05:50:11.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:11 smithi005 ceph-mon[84705]: from='client.64468 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:50:11.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:11 smithi005 ceph-mon[84705]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 615 B/s rd, 0 op/s 2024-03-22T05:50:11.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:11 smithi005 ceph-mon[89904]: from='client.64468 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:50:11.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:11 smithi005 ceph-mon[89904]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 615 B/s rd, 0 op/s 2024-03-22T05:50:13.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:13 smithi005 ceph-mon[89904]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:50:13.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:13 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:50:13.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:13 smithi005 ceph-mon[84705]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:50:13.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:13 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:50:13.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:13 smithi186 ceph-mon[67903]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:50:13.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:13 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:50:15.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:15 smithi005 ceph-mon[84705]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:50:15.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:15 smithi005 ceph-mon[84705]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:50:15.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:15 smithi005 ceph-mon[89904]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:50:15.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:15 smithi005 ceph-mon[89904]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:50:15.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:15 smithi186 ceph-mon[67903]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:50:15.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:15 smithi186 ceph-mon[67903]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:50:17.452 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:16 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:50:16] "GET /metrics HTTP/1.1" 200 38251 "" "Prometheus/2.43.0" 2024-03-22T05:50:17.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:17 smithi005 ceph-mon[89904]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:50:17.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:17 smithi005 ceph-mon[84705]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:50:17.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:17 smithi186 ceph-mon[67903]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:50:19.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:19 smithi005 ceph-mon[89904]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:50:19.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:19 smithi005 ceph-mon[84705]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:50:19.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:19 smithi186 ceph-mon[67903]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:50:21.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:21 smithi005 ceph-mon[89904]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:50:21.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:21 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-22T05:50:21.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:21 smithi005 ceph-mon[84705]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:50:21.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:21 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-22T05:50:21.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:21 smithi186 ceph-mon[67903]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:50:21.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:21 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-22T05:50:22.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:22 smithi186 ceph-mon[67903]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-22T05:50:22.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:22 smithi186 ceph-mon[67903]: Upgrade: osd.5 is safe to restart 2024-03-22T05:50:22.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:22 smithi005 ceph-mon[89904]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-22T05:50:22.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:22 smithi005 ceph-mon[89904]: Upgrade: osd.5 is safe to restart 2024-03-22T05:50:22.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:22 smithi005 ceph-mon[84705]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-22T05:50:22.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:22 smithi005 ceph-mon[84705]: Upgrade: osd.5 is safe to restart 2024-03-22T05:50:23.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:23 smithi005 ceph-mon[84705]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:50:23.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:23 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:50:23.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:23 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:50:23.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:23 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-22T05:50:23.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:23 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:50:23.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:23 smithi005 ceph-mon[89904]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:50:23.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:23 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:50:23.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:23 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:50:23.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:23 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-22T05:50:23.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:23 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:50:23.470 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:23 smithi186 ceph-mon[67903]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:50:23.470 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:23 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:50:23.470 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:23 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:50:23.471 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:23 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-22T05:50:23.471 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:23 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:50:24.442 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:24 smithi186 ceph-mon[67903]: Upgrade: Updating osd.5 2024-03-22T05:50:24.442 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:24 smithi186 ceph-mon[67903]: Deploying daemon osd.5 on smithi186 2024-03-22T05:50:24.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:24 smithi005 ceph-mon[89904]: Upgrade: Updating osd.5 2024-03-22T05:50:24.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:24 smithi005 ceph-mon[89904]: Deploying daemon osd.5 on smithi186 2024-03-22T05:50:24.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:24 smithi005 ceph-mon[84705]: Upgrade: Updating osd.5 2024-03-22T05:50:24.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:24 smithi005 ceph-mon[84705]: Deploying daemon osd.5 on smithi186 2024-03-22T05:50:25.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:25 smithi005 ceph-mon[89904]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:50:25.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:25 smithi005 ceph-mon[89904]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:50:25.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:25 smithi005 ceph-mon[84705]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:50:25.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:25 smithi005 ceph-mon[84705]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:50:25.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:25 smithi186 ceph-mon[67903]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:50:25.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:25 smithi186 ceph-mon[67903]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:50:26.294 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:50:26 smithi186 systemd[1]: Stopping Ceph osd.5 for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:50:26.794 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:50:26 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-5[32003]: 2024-03-22T05:50:26.337+0000 7f56231f2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T05:50:26.794 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:50:26 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-5[32003]: 2024-03-22T05:50:26.337+0000 7f56231f2700 -1 osd.5 130 *** Got signal Terminated *** 2024-03-22T05:50:26.794 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:50:26 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-5[32003]: 2024-03-22T05:50:26.337+0000 7f56231f2700 -1 osd.5 130 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-22T05:50:27.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:27 smithi005 ceph-mon[84705]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:50:27.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:27 smithi005 ceph-mon[84705]: osd.5 marked itself down and dead 2024-03-22T05:50:27.453 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:26 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:50:26] "GET /metrics HTTP/1.1" 200 38255 "" "Prometheus/2.43.0" 2024-03-22T05:50:27.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:27 smithi005 ceph-mon[89904]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:50:27.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:27 smithi005 ceph-mon[89904]: osd.5 marked itself down and dead 2024-03-22T05:50:27.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:27 smithi186 ceph-mon[67903]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:50:27.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:27 smithi186 ceph-mon[67903]: osd.5 marked itself down and dead 2024-03-22T05:50:27.986 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:50:27 smithi186 podman[82182]: 2024-03-22 05:50:27.681242003 +0000 UTC m=+1.494088382 container died f98c87f1a2cdf65178ddb50705f028c42de788c4033da5ab379d59c574102535 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-5, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, architecture=x86_64, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , distribution-scope=public, io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, name=centos-stream, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-03-22T05:50:27.986 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:50:27 smithi186 podman[82182]: 2024-03-22 05:50:27.986804884 +0000 UTC m=+1.799651259 container cleanup f98c87f1a2cdf65178ddb50705f028c42de788c4033da5ab379d59c574102535 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-5, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, architecture=x86_64, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, version=8, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, release=754, vendor=Red Hat, Inc., ceph=True, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux ) 2024-03-22T05:50:28.254 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:28 smithi186 ceph-mon[67903]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-22T05:50:28.254 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:28 smithi186 ceph-mon[67903]: osdmap e131: 8 total, 7 up, 8 in 2024-03-22T05:50:28.254 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:50:27 smithi186 bash[82182]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-5 2024-03-22T05:50:28.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:28 smithi005 ceph-mon[89904]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-22T05:50:28.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:28 smithi005 ceph-mon[89904]: osdmap e131: 8 total, 7 up, 8 in 2024-03-22T05:50:28.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:28 smithi005 ceph-mon[84705]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-22T05:50:28.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:28 smithi005 ceph-mon[84705]: osdmap e131: 8 total, 7 up, 8 in 2024-03-22T05:50:28.544 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:50:28 smithi186 podman[82210]: 2024-03-22 05:50:28.25500636 +0000 UTC m=+0.577587550 container remove f98c87f1a2cdf65178ddb50705f028c42de788c4033da5ab379d59c574102535 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-5, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , RELEASE=HEAD, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, release=754) 2024-03-22T05:50:28.891 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:50:28 smithi186 podman[82308]: 2024-03-22 05:50:28.496909869 +0000 UTC m=+0.018069088 image pull aa461e7c5e7f322b169c789b389d594b994ea7ffcbc9e24e563a3ec9b69bc3c1 quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9 2024-03-22T05:50:28.891 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:50:28 smithi186 podman[82308]: 2024-03-22 05:50:28.639549185 +0000 UTC m=+0.160708427 container create addb559d5488571599e1114209f1396480ef311fb2a19d737c81a580107d2346 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-5-deactivate, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-22T05:50:28.891 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:50:28 smithi186 podman[82308]: 2024-03-22 05:50:28.891857698 +0000 UTC m=+0.413016917 container init addb559d5488571599e1114209f1396480ef311fb2a19d737c81a580107d2346 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD) 2024-03-22T05:50:29.164 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:50:28 smithi186 podman[82308]: 2024-03-22 05:50:28.89515518 +0000 UTC m=+0.416314380 container start addb559d5488571599e1114209f1396480ef311fb2a19d737c81a580107d2346 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-5-deactivate, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217) 2024-03-22T05:50:29.164 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:50:28 smithi186 podman[82308]: 2024-03-22 05:50:28.966753208 +0000 UTC m=+0.487912421 container attach addb559d5488571599e1114209f1396480ef311fb2a19d737c81a580107d2346 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-5-deactivate, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-22T05:50:29.444 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:29 smithi186 ceph-mon[67903]: pgmap v220: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:50:29.444 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:29 smithi186 ceph-mon[67903]: osdmap e132: 8 total, 7 up, 8 in 2024-03-22T05:50:29.445 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:50:29 smithi186 podman[82308]: 2024-03-22 05:50:29.185480724 +0000 UTC m=+0.706639924 container died addb559d5488571599e1114209f1396480ef311fb2a19d737c81a580107d2346 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-5-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217) 2024-03-22T05:50:29.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:29 smithi005 ceph-mon[89904]: pgmap v220: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:50:29.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:29 smithi005 ceph-mon[89904]: osdmap e132: 8 total, 7 up, 8 in 2024-03-22T05:50:29.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:29 smithi005 ceph-mon[84705]: pgmap v220: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:50:29.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:29 smithi005 ceph-mon[84705]: osdmap e132: 8 total, 7 up, 8 in 2024-03-22T05:50:30.044 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:50:29 smithi186 podman[82334]: 2024-03-22 05:50:29.761698076 +0000 UTC m=+0.570162048 container remove addb559d5488571599e1114209f1396480ef311fb2a19d737c81a580107d2346 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True) 2024-03-22T05:50:30.044 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:50:29 smithi186 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@osd.5.service: Deactivated successfully. 2024-03-22T05:50:30.044 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:50:29 smithi186 systemd[1]: Stopped Ceph osd.5 for f59f36e2-e80a-11ee-95cd-87774f69a715. 2024-03-22T05:50:30.044 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:50:29 smithi186 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@osd.5.service: Consumed 16.018s CPU time. 2024-03-22T05:50:30.044 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:50:29 smithi186 systemd[1]: Starting Ceph osd.5 for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:50:30.334 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:30 smithi186 ceph-mon[67903]: Health check failed: Degraded data redundancy: 49/723 objects degraded (6.777%), 13 pgs degraded (PG_DEGRADED) 2024-03-22T05:50:30.334 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:50:30 smithi186 podman[82440]: 2024-03-22 05:50:30.094712886 +0000 UTC m=+0.019028583 image pull aa461e7c5e7f322b169c789b389d594b994ea7ffcbc9e24e563a3ec9b69bc3c1 quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9 2024-03-22T05:50:30.334 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:50:30 smithi186 podman[82440]: 2024-03-22 05:50:30.229425582 +0000 UTC m=+0.153741258 container create 57b04773a4d7e839d2bbab5b3956d4598823f53fd1362036fc2f3ac03e28e4aa (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-5-activate, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-22T05:50:30.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:30 smithi005 ceph-mon[89904]: Health check failed: Degraded data redundancy: 49/723 objects degraded (6.777%), 13 pgs degraded (PG_DEGRADED) 2024-03-22T05:50:30.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:30 smithi005 ceph-mon[84705]: Health check failed: Degraded data redundancy: 49/723 objects degraded (6.777%), 13 pgs degraded (PG_DEGRADED) 2024-03-22T05:50:30.793 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:50:30 smithi186 podman[82440]: 2024-03-22 05:50:30.457113373 +0000 UTC m=+0.381429059 container init 57b04773a4d7e839d2bbab5b3956d4598823f53fd1362036fc2f3ac03e28e4aa (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-5-activate, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319) 2024-03-22T05:50:30.794 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:50:30 smithi186 podman[82440]: 2024-03-22 05:50:30.460409848 +0000 UTC m=+0.384725538 container start 57b04773a4d7e839d2bbab5b3956d4598823f53fd1362036fc2f3ac03e28e4aa (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-5-activate, org.label-schema.build-date=20240319, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-22T05:50:30.794 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:50:30 smithi186 podman[82440]: 2024-03-22 05:50:30.515318264 +0000 UTC m=+0.439633956 container attach 57b04773a4d7e839d2bbab5b3956d4598823f53fd1362036fc2f3ac03e28e4aa (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-5-activate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-22T05:50:31.210 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:50:30 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-5-activate[82455]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-22T05:50:31.210 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:50:30 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-5-activate[82455]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-03-22T05:50:31.210 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:50:30 smithi186 bash[82440]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-22T05:50:31.210 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:50:30 smithi186 bash[82440]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-03-22T05:50:31.210 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:50:30 smithi186 bash[82440]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-03-22T05:50:31.210 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:50:30 smithi186 bash[82440]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-03-22T05:50:31.210 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:50:30 smithi186 bash[82440]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-03-22T05:50:31.210 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:50:30 smithi186 bash[82440]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-22T05:50:31.211 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:50:30 smithi186 bash[82440]: --> ceph-volume raw activate successful for osd ID: 5 2024-03-22T05:50:31.211 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:50:30 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-5-activate[82455]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-03-22T05:50:31.211 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:50:30 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-5-activate[82455]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-03-22T05:50:31.211 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:50:30 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-5-activate[82455]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-03-22T05:50:31.211 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:50:30 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-5-activate[82455]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-22T05:50:31.211 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:50:30 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-5-activate[82455]: --> ceph-volume raw activate successful for osd ID: 5 2024-03-22T05:50:31.211 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:50:30 smithi186 podman[82440]: 2024-03-22 05:50:30.929078799 +0000 UTC m=+0.853394484 container died 57b04773a4d7e839d2bbab5b3956d4598823f53fd1362036fc2f3ac03e28e4aa (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-5-activate, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-22T05:50:31.509 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:31 smithi186 ceph-mon[67903]: pgmap v222: 161 pgs: 13 active+undersized+degraded, 6 stale+active+clean, 29 active+undersized, 113 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 49/723 objects degraded (6.777%) 2024-03-22T05:50:31.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:31 smithi005 ceph-mon[89904]: pgmap v222: 161 pgs: 13 active+undersized+degraded, 6 stale+active+clean, 29 active+undersized, 113 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 49/723 objects degraded (6.777%) 2024-03-22T05:50:31.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:31 smithi005 ceph-mon[84705]: pgmap v222: 161 pgs: 13 active+undersized+degraded, 6 stale+active+clean, 29 active+undersized, 113 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 49/723 objects degraded (6.777%) 2024-03-22T05:50:31.794 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:50:31 smithi186 podman[82578]: 2024-03-22 05:50:31.510359908 +0000 UTC m=+0.576388550 container remove 57b04773a4d7e839d2bbab5b3956d4598823f53fd1362036fc2f3ac03e28e4aa (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-5-activate, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-22T05:50:31.794 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:50:31 smithi186 podman[82644]: 2024-03-22 05:50:31.635664245 +0000 UTC m=+0.018293937 image pull aa461e7c5e7f322b169c789b389d594b994ea7ffcbc9e24e563a3ec9b69bc3c1 quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9 2024-03-22T05:50:31.794 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:50:31 smithi186 podman[82644]: 2024-03-22 05:50:31.736282259 +0000 UTC m=+0.118929006 container create b80f7eceb2d771db9d94f7071eb90160cd2fcf2524b499f5f8733a2f5fbb755d (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217) 2024-03-22T05:50:32.294 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:50:31 smithi186 podman[82644]: 2024-03-22 05:50:31.963513792 +0000 UTC m=+0.346143482 container init b80f7eceb2d771db9d94f7071eb90160cd2fcf2524b499f5f8733a2f5fbb755d (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-22T05:50:32.294 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:50:31 smithi186 podman[82644]: 2024-03-22 05:50:31.966805561 +0000 UTC m=+0.349435253 container start b80f7eceb2d771db9d94f7071eb90160cd2fcf2524b499f5f8733a2f5fbb755d (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319) 2024-03-22T05:50:32.294 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:50:31 smithi186 ceph-osd[82664]: -- 172.21.15.186:0/1883361799 <== mon.2 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55fb9b4bd680 con 0x55fb9b486c00 2024-03-22T05:50:32.294 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:50:32 smithi186 bash[82644]: b80f7eceb2d771db9d94f7071eb90160cd2fcf2524b499f5f8733a2f5fbb755d 2024-03-22T05:50:32.294 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:50:32 smithi186 systemd[1]: Started Ceph osd.5 for f59f36e2-e80a-11ee-95cd-87774f69a715. 2024-03-22T05:50:32.793 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:50:32 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-5[82660]: 2024-03-22T05:50:32.512+0000 7fadaca787c0 -1 Falling back to public interface 2024-03-22T05:50:33.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:33 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:50:33.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:33 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:50:33.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:33 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:50:33.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:33 smithi186 ceph-mon[67903]: pgmap v223: 161 pgs: 22 active+undersized+degraded, 37 active+undersized, 102 active+clean; 585 KiB data, 194 MiB used, 715 GiB / 715 GiB avail; 81/723 objects degraded (11.203%) 2024-03-22T05:50:33.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:33 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:50:33.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:33 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:50:33.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:33 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:50:33.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:33 smithi005 ceph-mon[89904]: pgmap v223: 161 pgs: 22 active+undersized+degraded, 37 active+undersized, 102 active+clean; 585 KiB data, 194 MiB used, 715 GiB / 715 GiB avail; 81/723 objects degraded (11.203%) 2024-03-22T05:50:33.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:33 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:50:33.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:33 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:50:33.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:33 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:50:33.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:33 smithi005 ceph-mon[84705]: pgmap v223: 161 pgs: 22 active+undersized+degraded, 37 active+undersized, 102 active+clean; 585 KiB data, 194 MiB used, 715 GiB / 715 GiB avail; 81/723 objects degraded (11.203%) 2024-03-22T05:50:35.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:35 smithi005 ceph-mon[89904]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:50:35.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:35 smithi005 ceph-mon[89904]: pgmap v224: 161 pgs: 22 active+undersized+degraded, 37 active+undersized, 102 active+clean; 585 KiB data, 194 MiB used, 715 GiB / 715 GiB avail; 81/723 objects degraded (11.203%) 2024-03-22T05:50:35.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:35 smithi005 ceph-mon[84705]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:50:35.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:35 smithi005 ceph-mon[84705]: pgmap v224: 161 pgs: 22 active+undersized+degraded, 37 active+undersized, 102 active+clean; 585 KiB data, 194 MiB used, 715 GiB / 715 GiB avail; 81/723 objects degraded (11.203%) 2024-03-22T05:50:35.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:35 smithi186 ceph-mon[67903]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:50:35.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:35 smithi186 ceph-mon[67903]: pgmap v224: 161 pgs: 22 active+undersized+degraded, 37 active+undersized, 102 active+clean; 585 KiB data, 194 MiB used, 715 GiB / 715 GiB avail; 81/723 objects degraded (11.203%) 2024-03-22T05:50:35.544 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:50:35 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-5[82660]: 2024-03-22T05:50:35.231+0000 7fadaca787c0 -1 osd.5 0 read_superblock omap replica is missing. 2024-03-22T05:50:35.544 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:50:35 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-5[82660]: 2024-03-22T05:50:35.257+0000 7fadaca787c0 -1 osd.5 130 log_to_monitors true 2024-03-22T05:50:36.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:36 smithi005 ceph-mon[89904]: from='osd.5 [v2:172.21.15.186:6808/783268671,v1:172.21.15.186:6809/783268671]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-22T05:50:36.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:36 smithi005 ceph-mon[89904]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-22T05:50:36.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:36 smithi005 ceph-mon[84705]: from='osd.5 [v2:172.21.15.186:6808/783268671,v1:172.21.15.186:6809/783268671]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-22T05:50:36.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:36 smithi005 ceph-mon[84705]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-22T05:50:36.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:36 smithi186 ceph-mon[67903]: from='osd.5 [v2:172.21.15.186:6808/783268671,v1:172.21.15.186:6809/783268671]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-22T05:50:36.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:36 smithi186 ceph-mon[67903]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-22T05:50:37.043 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:50:36 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-5[82660]: 2024-03-22T05:50:36.667+0000 7fad99ec8700 -1 osd.5 130 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-22T05:50:37.452 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:36 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:50:36] "GET /metrics HTTP/1.1" 200 38255 "" "Prometheus/2.43.0" 2024-03-22T05:50:37.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:37 smithi005 ceph-mon[89904]: pgmap v225: 161 pgs: 22 active+undersized+degraded, 37 active+undersized, 102 active+clean; 585 KiB data, 194 MiB used, 715 GiB / 715 GiB avail; 81/723 objects degraded (11.203%) 2024-03-22T05:50:37.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:37 smithi005 ceph-mon[89904]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-22T05:50:37.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:37 smithi005 ceph-mon[89904]: from='osd.5 [v2:172.21.15.186:6808/783268671,v1:172.21.15.186:6809/783268671]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-03-22T05:50:37.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:37 smithi005 ceph-mon[89904]: osdmap e133: 8 total, 7 up, 8 in 2024-03-22T05:50:37.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:37 smithi005 ceph-mon[89904]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-03-22T05:50:37.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:37 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:50:37.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:37 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:50:37.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:37 smithi005 ceph-mon[89904]: from='osd.5 ' entity='osd.5' 2024-03-22T05:50:37.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:37 smithi005 ceph-mon[84705]: pgmap v225: 161 pgs: 22 active+undersized+degraded, 37 active+undersized, 102 active+clean; 585 KiB data, 194 MiB used, 715 GiB / 715 GiB avail; 81/723 objects degraded (11.203%) 2024-03-22T05:50:37.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:37 smithi005 ceph-mon[84705]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-22T05:50:37.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:37 smithi005 ceph-mon[84705]: from='osd.5 [v2:172.21.15.186:6808/783268671,v1:172.21.15.186:6809/783268671]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-03-22T05:50:37.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:37 smithi005 ceph-mon[84705]: osdmap e133: 8 total, 7 up, 8 in 2024-03-22T05:50:37.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:37 smithi005 ceph-mon[84705]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-03-22T05:50:37.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:37 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:50:37.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:37 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:50:37.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:37 smithi005 ceph-mon[84705]: from='osd.5 ' entity='osd.5' 2024-03-22T05:50:37.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:37 smithi186 ceph-mon[67903]: pgmap v225: 161 pgs: 22 active+undersized+degraded, 37 active+undersized, 102 active+clean; 585 KiB data, 194 MiB used, 715 GiB / 715 GiB avail; 81/723 objects degraded (11.203%) 2024-03-22T05:50:37.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:37 smithi186 ceph-mon[67903]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-22T05:50:37.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:37 smithi186 ceph-mon[67903]: from='osd.5 [v2:172.21.15.186:6808/783268671,v1:172.21.15.186:6809/783268671]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-03-22T05:50:37.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:37 smithi186 ceph-mon[67903]: osdmap e133: 8 total, 7 up, 8 in 2024-03-22T05:50:37.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:37 smithi186 ceph-mon[67903]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-03-22T05:50:37.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:37 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:50:37.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:37 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:50:37.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:37 smithi186 ceph-mon[67903]: from='osd.5 ' entity='osd.5' 2024-03-22T05:50:38.439 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:38 smithi186 ceph-mon[67903]: Health check update: Degraded data redundancy: 81/723 objects degraded (11.203%), 22 pgs degraded (PG_DEGRADED) 2024-03-22T05:50:38.439 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:38 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:50:38.439 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:38 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:50:38.439 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:38 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:50:38.439 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:38 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:50:38.440 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:38 smithi186 ceph-mon[67903]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-22T05:50:38.440 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:38 smithi186 ceph-mon[67903]: osd.5 [v2:172.21.15.186:6808/783268671,v1:172.21.15.186:6809/783268671] boot 2024-03-22T05:50:38.440 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:38 smithi186 ceph-mon[67903]: osdmap e134: 8 total, 8 up, 8 in 2024-03-22T05:50:38.440 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:38 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T05:50:38.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:38 smithi005 ceph-mon[84705]: Health check update: Degraded data redundancy: 81/723 objects degraded (11.203%), 22 pgs degraded (PG_DEGRADED) 2024-03-22T05:50:38.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:38 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:50:38.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:38 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:50:38.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:38 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:50:38.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:38 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:50:38.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:38 smithi005 ceph-mon[84705]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-22T05:50:38.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:38 smithi005 ceph-mon[84705]: osd.5 [v2:172.21.15.186:6808/783268671,v1:172.21.15.186:6809/783268671] boot 2024-03-22T05:50:38.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:38 smithi005 ceph-mon[84705]: osdmap e134: 8 total, 8 up, 8 in 2024-03-22T05:50:38.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:38 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T05:50:38.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:38 smithi005 ceph-mon[89904]: Health check update: Degraded data redundancy: 81/723 objects degraded (11.203%), 22 pgs degraded (PG_DEGRADED) 2024-03-22T05:50:38.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:38 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:50:38.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:38 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:50:38.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:38 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:50:38.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:38 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:50:38.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:38 smithi005 ceph-mon[89904]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-22T05:50:38.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:38 smithi005 ceph-mon[89904]: osd.5 [v2:172.21.15.186:6808/783268671,v1:172.21.15.186:6809/783268671] boot 2024-03-22T05:50:38.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:38 smithi005 ceph-mon[89904]: osdmap e134: 8 total, 8 up, 8 in 2024-03-22T05:50:38.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:38 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-22T05:50:39.381 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:39 smithi186 ceph-mon[67903]: pgmap v228: 161 pgs: 22 active+undersized+degraded, 37 active+undersized, 102 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 81/723 objects degraded (11.203%) 2024-03-22T05:50:39.381 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:39 smithi186 ceph-mon[67903]: osdmap e135: 8 total, 8 up, 8 in 2024-03-22T05:50:39.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:39 smithi005 ceph-mon[84705]: pgmap v228: 161 pgs: 22 active+undersized+degraded, 37 active+undersized, 102 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 81/723 objects degraded (11.203%) 2024-03-22T05:50:39.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:39 smithi005 ceph-mon[84705]: osdmap e135: 8 total, 8 up, 8 in 2024-03-22T05:50:39.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:39 smithi005 ceph-mon[89904]: pgmap v228: 161 pgs: 22 active+undersized+degraded, 37 active+undersized, 102 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 81/723 objects degraded (11.203%) 2024-03-22T05:50:39.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:39 smithi005 ceph-mon[89904]: osdmap e135: 8 total, 8 up, 8 in 2024-03-22T05:50:40.414 INFO:teuthology.orchestra.run.smithi005.stdout:true 2024-03-22T05:50:41.093 INFO:teuthology.orchestra.run.smithi005.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T05:50:41.093 INFO:teuthology.orchestra.run.smithi005.stdout:alertmanager.a smithi005 *:9093,9094 running (8m) 78s ago 30m 28.0M - 0.25.0 c8568f914cd2 ee99dfc435e8 2024-03-22T05:50:41.093 INFO:teuthology.orchestra.run.smithi005.stdout:grafana.a smithi186 *:3000 running (7m) 4s ago 29m 81.6M - 9.4.12 2bacad6d85d8 16fd57dbbd71 2024-03-22T05:50:41.093 INFO:teuthology.orchestra.run.smithi005.stdout:iscsi.foo.smithi005.mcdwbp smithi005 unknown 78s ago 29m 64.6M - 3.5 e1d6a67b021e 5215e03bf81b 2024-03-22T05:50:41.093 INFO:teuthology.orchestra.run.smithi005.stdout:mgr.x smithi186 *:8443,9283,8765 running (9m) 4s ago 34m 453M - 19.0.0-2299-gea01217f aa461e7c5e7f c556421a73f3 2024-03-22T05:50:41.093 INFO:teuthology.orchestra.run.smithi005.stdout:mgr.y smithi005 *:8443,9283,8765 running (19m) 78s ago 36m 511M - 19.0.0-2299-gea01217f aa461e7c5e7f 62a813eebc0d 2024-03-22T05:50:41.093 INFO:teuthology.orchestra.run.smithi005.stdout:mon.a smithi005 running (5m) 78s ago 36m 55.3M 2048M 19.0.0-2299-gea01217f aa461e7c5e7f 111a2aa24dd9 2024-03-22T05:50:41.093 INFO:teuthology.orchestra.run.smithi005.stdout:mon.b smithi186 running (6m) 4s ago 34m 51.5M 2048M 19.0.0-2299-gea01217f aa461e7c5e7f c1798feac15b 2024-03-22T05:50:41.093 INFO:teuthology.orchestra.run.smithi005.stdout:mon.c smithi005 running (5m) 78s ago 35m 42.0M 2048M 19.0.0-2299-gea01217f aa461e7c5e7f 708c72ff74f5 2024-03-22T05:50:41.093 INFO:teuthology.orchestra.run.smithi005.stdout:node-exporter.a smithi005 *:9100 running (8m) 78s ago 30m 16.8M - 1.5.0 0da6a335fe13 f635cd72b9d3 2024-03-22T05:50:41.093 INFO:teuthology.orchestra.run.smithi005.stdout:node-exporter.b smithi186 *:9100 running (8m) 4s ago 30m 15.0M - 1.5.0 0da6a335fe13 d7e0b3d20191 2024-03-22T05:50:41.093 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0 smithi005 running (3m) 78s ago 34m 44.8M 2993M 19.0.0-2299-gea01217f aa461e7c5e7f 31b6cdee5f5b 2024-03-22T05:50:41.093 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1 smithi005 running (3m) 78s ago 33m 44.2M 2993M 19.0.0-2299-gea01217f aa461e7c5e7f 7d21bc9fd3c2 2024-03-22T05:50:41.093 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2 smithi005 running (2m) 78s ago 33m 42.0M 2993M 19.0.0-2299-gea01217f aa461e7c5e7f 628204cb7c98 2024-03-22T05:50:41.094 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3 smithi005 running (83s) 78s ago 33m 14.9M 2993M 19.0.0-2299-gea01217f aa461e7c5e7f d31a01a0230c 2024-03-22T05:50:41.094 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4 smithi186 running (46s) 4s ago 32m 54.0M 3248M 19.0.0-2299-gea01217f aa461e7c5e7f 8f7619b726f3 2024-03-22T05:50:41.094 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5 smithi186 running (9s) 4s ago 32m 12.0M 3248M 19.0.0-2299-gea01217f aa461e7c5e7f b80f7eceb2d7 2024-03-22T05:50:41.094 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6 smithi186 running (32m) 4s ago 32m 56.2M 3248M 17.2.0 e1d6a67b021e 704ac49c68c4 2024-03-22T05:50:41.094 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7 smithi186 running (31m) 4s ago 31m 59.3M 3248M 17.2.0 e1d6a67b021e f415b90edf63 2024-03-22T05:50:41.094 INFO:teuthology.orchestra.run.smithi005.stdout:prometheus.a smithi186 *:9095 running (8m) 4s ago 30m 72.0M - 2.43.0 a07b618ecd1d a5a043588d6d 2024-03-22T05:50:41.094 INFO:teuthology.orchestra.run.smithi005.stdout:rgw.foo.smithi005.exdhqu smithi005 *:8000 running (29m) 78s ago 29m 81.5M - 17.2.0 e1d6a67b021e 6bb27fe11553 2024-03-22T05:50:41.094 INFO:teuthology.orchestra.run.smithi005.stdout:rgw.foo.smithi186.mcgodk smithi186 *:8000 running (29m) 4s ago 29m 79.9M - 17.2.0 e1d6a67b021e 19b5c20a34f7 2024-03-22T05:50:41.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:41 smithi005 ceph-mon[89904]: pgmap v230: 161 pgs: 19 peering, 14 active+undersized+degraded, 18 active+undersized, 110 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 54/723 objects degraded (7.469%) 2024-03-22T05:50:41.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:41 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:50:41.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:41 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:50:41.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:41 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:50:41.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:41 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:50:41.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:41 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:50:41.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:41 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:50:41.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:41 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:50:41.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:41 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:50:41.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:41 smithi005 ceph-mon[84705]: pgmap v230: 161 pgs: 19 peering, 14 active+undersized+degraded, 18 active+undersized, 110 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 54/723 objects degraded (7.469%) 2024-03-22T05:50:41.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:41 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:50:41.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:41 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:50:41.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:41 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:50:41.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:41 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:50:41.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:41 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:50:41.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:41 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:50:41.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:41 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:50:41.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:41 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:50:41.523 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-03-22T05:50:41.523 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": { 2024-03-22T05:50:41.523 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev)": 3 2024-03-22T05:50:41.523 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:50:41.523 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr": { 2024-03-22T05:50:41.523 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev)": 2 2024-03-22T05:50:41.524 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:50:41.524 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": { 2024-03-22T05:50:41.524 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-03-22T05:50:41.524 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev)": 6 2024-03-22T05:50:41.524 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:50:41.524 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw": { 2024-03-22T05:50:41.524 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-22T05:50:41.524 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:50:41.524 INFO:teuthology.orchestra.run.smithi005.stdout: "overall": { 2024-03-22T05:50:41.524 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2024-03-22T05:50:41.524 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev)": 11 2024-03-22T05:50:41.524 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-03-22T05:50:41.524 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-03-22T05:50:41.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:41 smithi186 ceph-mon[67903]: pgmap v230: 161 pgs: 19 peering, 14 active+undersized+degraded, 18 active+undersized, 110 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 54/723 objects degraded (7.469%) 2024-03-22T05:50:41.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:41 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:50:41.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:41 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:50:41.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:41 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:50:41.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:41 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:50:41.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:41 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:50:41.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:41 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:50:41.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:41 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:50:41.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:41 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:50:41.875 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-03-22T05:50:41.876 INFO:teuthology.orchestra.run.smithi005.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9", 2024-03-22T05:50:41.876 INFO:teuthology.orchestra.run.smithi005.stdout: "in_progress": true, 2024-03-22T05:50:41.876 INFO:teuthology.orchestra.run.smithi005.stdout: "which": "Upgrading daemons of type(s) crash,osd", 2024-03-22T05:50:41.876 INFO:teuthology.orchestra.run.smithi005.stdout: "services_complete": [], 2024-03-22T05:50:41.876 INFO:teuthology.orchestra.run.smithi005.stdout: "progress": "6/8 daemons upgraded", 2024-03-22T05:50:41.876 INFO:teuthology.orchestra.run.smithi005.stdout: "message": "Currently upgrading osd daemons", 2024-03-22T05:50:41.876 INFO:teuthology.orchestra.run.smithi005.stdout: "is_paused": false 2024-03-22T05:50:41.876 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-03-22T05:50:42.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:42 smithi005 ceph-mon[89904]: from='client.54543 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:50:42.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:42 smithi005 ceph-mon[89904]: from='client.64483 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:50:42.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:42 smithi005 ceph-mon[89904]: pgmap v231: 161 pgs: 19 peering, 14 active+undersized+degraded, 18 active+undersized, 110 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 54/723 objects degraded (7.469%) 2024-03-22T05:50:42.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:42 smithi005 ceph-mon[89904]: pgmap v232: 161 pgs: 19 peering, 14 active+undersized+degraded, 18 active+undersized, 110 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 54/723 objects degraded (7.469%) 2024-03-22T05:50:42.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:42 smithi005 ceph-mon[89904]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:50:42.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:42 smithi005 ceph-mon[89904]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:50:42.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:42 smithi005 ceph-mon[89904]: from='client.54555 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:50:42.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:42 smithi005 ceph-mon[89904]: from='client.? 172.21.15.5:0/1648435222' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:50:42.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:42 smithi005 ceph-mon[84705]: from='client.54543 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:50:42.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:42 smithi005 ceph-mon[84705]: from='client.64483 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:50:42.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:42 smithi005 ceph-mon[84705]: pgmap v231: 161 pgs: 19 peering, 14 active+undersized+degraded, 18 active+undersized, 110 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 54/723 objects degraded (7.469%) 2024-03-22T05:50:42.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:42 smithi005 ceph-mon[84705]: pgmap v232: 161 pgs: 19 peering, 14 active+undersized+degraded, 18 active+undersized, 110 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 54/723 objects degraded (7.469%) 2024-03-22T05:50:42.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:42 smithi005 ceph-mon[84705]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:50:42.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:42 smithi005 ceph-mon[84705]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:50:42.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:42 smithi005 ceph-mon[84705]: from='client.54555 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:50:42.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:42 smithi005 ceph-mon[84705]: from='client.? 172.21.15.5:0/1648435222' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:50:42.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:42 smithi186 ceph-mon[67903]: from='client.54543 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:50:42.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:42 smithi186 ceph-mon[67903]: from='client.64483 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:50:42.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:42 smithi186 ceph-mon[67903]: pgmap v231: 161 pgs: 19 peering, 14 active+undersized+degraded, 18 active+undersized, 110 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 54/723 objects degraded (7.469%) 2024-03-22T05:50:42.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:42 smithi186 ceph-mon[67903]: pgmap v232: 161 pgs: 19 peering, 14 active+undersized+degraded, 18 active+undersized, 110 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 54/723 objects degraded (7.469%) 2024-03-22T05:50:42.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:42 smithi186 ceph-mon[67903]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:50:42.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:42 smithi186 ceph-mon[67903]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:50:42.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:42 smithi186 ceph-mon[67903]: from='client.54555 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:50:42.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:42 smithi186 ceph-mon[67903]: from='client.? 172.21.15.5:0/1648435222' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:50:42.893 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:42 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:50:42.889+0000 7f7f0ecc9700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:50:42.893 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:42 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:50:42.894 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:42 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:50:42.894 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:42 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:50:42.894 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:42 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:50:42.894 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:42 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:50:42.894 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:42 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:50:42.894 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:42 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:50:42.894 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:42 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:50:42.894 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:42 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:50:42.894 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:42 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:50:42.894 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:42 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:50:42.894 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:42 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:50:42.894 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:42 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:50:42.894 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:42 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:50:42.895 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:42 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:50:42.895 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:42 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:50:42.895 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:42 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:50:42.895 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:42 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:50:42.895 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:42 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:50:42.895 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:42 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:50:42.895 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:42 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:50:42.896 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:42 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:50:42.896 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:42 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:50:42.896 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:42 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:50:42.896 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:42 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:50:42.896 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:42 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:50:42.896 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:42 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:50:42.896 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:42 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:50:42.896 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:42 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return future.result(timeout) 2024-03-22T05:50:42.896 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:42 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:50:42.896 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:42 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.__get_result() 2024-03-22T05:50:42.896 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:42 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:50:42.896 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:42 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: raise self._exception 2024-03-22T05:50:42.896 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:42 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:50:42.896 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:42 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: config_blobs=daemon_spec.final_config, 2024-03-22T05:50:42.897 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:42 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:50:42.897 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:42 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:50:42.897 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:42 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:50:42.897 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:42 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:50:42.897 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:42 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:50:42.897 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:42 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:50:42.897 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:42 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:50:42.897 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:42 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:50:42.897 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:42 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:50:42.897 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:42 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:50:42.897 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:42 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:50:42.897 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:42 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:50:42.897 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:42 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:50:42.897 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:42 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:50:42.898 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:42 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:50:42.898 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:42 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:50:42.898 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:42 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:50:42.898 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:42 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:50:42.898 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:42 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:50:42.898 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:42 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:50:42.898 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:42 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:50:42.898 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:42 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:50:42.898 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:42 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:50:42.952 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:42 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:50:42.903+0000 7f7f3e571700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-03-22T05:50:43.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:43 smithi005 ceph-mon[84705]: from='client.54567 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:50:43.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:43 smithi005 ceph-mon[84705]: Health check update: Degraded data redundancy: 54/723 objects degraded (7.469%), 14 pgs degraded (PG_DEGRADED) 2024-03-22T05:50:43.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:43 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:50:43.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:43 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:50:43.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:43 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:50:43.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:43 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:50:43.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:43 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-22T05:50:43.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:43 smithi005 ceph-mon[89904]: from='client.54567 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:50:43.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:43 smithi005 ceph-mon[89904]: Health check update: Degraded data redundancy: 54/723 objects degraded (7.469%), 14 pgs degraded (PG_DEGRADED) 2024-03-22T05:50:43.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:43 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:50:43.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:43 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:50:43.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:43 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:50:43.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:43 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:50:43.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:43 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-22T05:50:43.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:43 smithi186 ceph-mon[67903]: from='client.54567 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:50:43.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:43 smithi186 ceph-mon[67903]: Health check update: Degraded data redundancy: 54/723 objects degraded (7.469%), 14 pgs degraded (PG_DEGRADED) 2024-03-22T05:50:43.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:43 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:50:43.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:43 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:50:43.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:43 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:50:43.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:43 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:50:43.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:43 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-22T05:50:44.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:44 smithi005 ceph-mon[84705]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:50:44.453 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:50:44.453 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:50:44.453 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:50:44.453 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:50:44.453 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:50:44.453 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:50:44.453 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:50:44.453 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:50:44.454 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:50:44.454 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:50:44.454 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:50:44.454 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:50:44.454 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:50:44.454 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:50:44.454 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:50:44.454 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:50:44.454 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:50:44.454 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:50:44.454 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:50:44.454 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:50:44.454 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:50:44.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:50:44.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:50:44.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:50:44.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:50:44.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:50:44.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:50:44.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:50:44.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: return future.result(timeout) 2024-03-22T05:50:44.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:50:44.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.__get_result() 2024-03-22T05:50:44.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:50:44.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: raise self._exception 2024-03-22T05:50:44.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:50:44.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:50:44.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:50:44.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:50:44.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:50:44.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:50:44.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:50:44.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:50:44.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:50:44.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:50:44.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:50:44.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:50:44.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:50:44.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:50:44.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:50:44.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:50:44.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:50:44.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:50:44.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:50:44.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:50:44.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:50:44.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:50:44.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:50:44.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:50:44.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:50:44.457 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:44 smithi005 ceph-mon[84705]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-22T05:50:44.457 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:44 smithi005 ceph-mon[84705]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-03-22T05:50:44.457 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:44 smithi005 ceph-mon[84705]: pgmap v233: 161 pgs: 19 peering, 142 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 780 B/s rd, 0 op/s 2024-03-22T05:50:44.457 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:44 smithi005 ceph-mon[84705]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 54/723 objects degraded (7.469%), 14 pgs degraded) 2024-03-22T05:50:44.458 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:44 smithi005 ceph-mon[89904]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:50:44.458 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:50:44.458 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:50:44.458 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:50:44.458 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:50:44.458 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:50:44.458 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:50:44.458 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:50:44.458 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:50:44.458 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:50:44.458 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:50:44.458 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:50:44.458 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:50:44.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:50:44.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:50:44.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:50:44.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:50:44.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:50:44.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:50:44.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:50:44.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:50:44.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:50:44.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:50:44.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:50:44.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:50:44.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:50:44.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:50:44.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:50:44.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:50:44.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: return future.result(timeout) 2024-03-22T05:50:44.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:50:44.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.__get_result() 2024-03-22T05:50:44.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:50:44.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: raise self._exception 2024-03-22T05:50:44.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:50:44.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:50:44.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:50:44.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:50:44.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:50:44.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:50:44.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:50:44.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:50:44.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:50:44.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:50:44.461 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:50:44.461 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:50:44.461 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:50:44.461 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:50:44.461 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:50:44.461 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:50:44.461 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:50:44.461 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:50:44.461 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:50:44.461 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:50:44.461 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:50:44.461 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:50:44.461 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:50:44.462 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:50:44.462 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:50:44.462 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:44 smithi005 ceph-mon[89904]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-22T05:50:44.462 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:44 smithi005 ceph-mon[89904]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-03-22T05:50:44.462 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:44 smithi005 ceph-mon[89904]: pgmap v233: 161 pgs: 19 peering, 142 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 780 B/s rd, 0 op/s 2024-03-22T05:50:44.462 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:44 smithi005 ceph-mon[89904]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 54/723 objects degraded (7.469%), 14 pgs degraded) 2024-03-22T05:50:44.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:44 smithi186 ceph-mon[67903]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:50:44.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:50:44.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:50:44.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:50:44.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:50:44.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:50:44.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:50:44.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:50:44.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:50:44.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:50:44.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:50:44.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:50:44.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:50:44.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:50:44.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:50:44.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:50:44.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:50:44.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:50:44.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:50:44.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:50:44.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:50:44.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:50:44.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:50:44.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:50:44.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:50:44.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:50:44.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:50:44.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:50:44.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:50:44.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: return future.result(timeout) 2024-03-22T05:50:44.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:50:44.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.__get_result() 2024-03-22T05:50:44.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:50:44.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: raise self._exception 2024-03-22T05:50:44.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:50:44.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:50:44.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:50:44.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:50:44.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:50:44.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:50:44.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:50:44.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:50:44.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:50:44.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:50:44.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:50:44.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:50:44.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:50:44.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:50:44.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:50:44.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:50:44.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:50:44.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:50:44.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:50:44.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:50:44.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:50:44.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:50:44.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:50:44.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:50:44.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:50:44.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:44 smithi186 ceph-mon[67903]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-22T05:50:44.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:44 smithi186 ceph-mon[67903]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-03-22T05:50:44.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:44 smithi186 ceph-mon[67903]: pgmap v233: 161 pgs: 19 peering, 142 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 780 B/s rd, 0 op/s 2024-03-22T05:50:44.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:44 smithi186 ceph-mon[67903]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 54/723 objects degraded (7.469%), 14 pgs degraded) 2024-03-22T05:50:45.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:45 smithi186 ceph-mon[67903]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:50:45.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:45 smithi005 ceph-mon[84705]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:50:45.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:45 smithi005 ceph-mon[89904]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:50:46.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:46 smithi186 ceph-mon[67903]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 603 B/s rd, 0 op/s 2024-03-22T05:50:46.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:46 smithi005 ceph-mon[89904]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 603 B/s rd, 0 op/s 2024-03-22T05:50:46.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:46 smithi005 ceph-mon[84705]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 603 B/s rd, 0 op/s 2024-03-22T05:50:47.452 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:46 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:50:46] "GET /metrics HTTP/1.1" 200 38113 "" "Prometheus/2.43.0" 2024-03-22T05:50:48.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:48 smithi186 ceph-mon[67903]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:50:48.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:48 smithi005 ceph-mon[89904]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:50:48.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:48 smithi005 ceph-mon[84705]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:50:50.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:50 smithi186 ceph-mon[67903]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-22T05:50:50.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:50 smithi005 ceph-mon[89904]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-22T05:50:50.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:50 smithi005 ceph-mon[84705]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-22T05:50:52.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:52 smithi186 ceph-mon[67903]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-22T05:50:52.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:52 smithi005 ceph-mon[89904]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-22T05:50:52.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:52 smithi005 ceph-mon[84705]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-22T05:50:53.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:53 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:50:53.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:53 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:50:53.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:53 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:50:53.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:53 smithi186 ceph-mon[67903]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:50:53.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:53 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:50:53.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:53 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:50:53.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:53 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:50:53.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:53 smithi005 ceph-mon[84705]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:50:53.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:53 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:50:53.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:53 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:50:53.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:53 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:50:53.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:53 smithi005 ceph-mon[89904]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:50:54.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:54 smithi005 ceph-mon[84705]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:50:54.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:54 smithi005 ceph-mon[89904]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:50:55.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:54 smithi186 ceph-mon[67903]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:50:55.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:55 smithi005 ceph-mon[89904]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-22T05:50:55.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:55 smithi005 ceph-mon[84705]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-22T05:50:56.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:55 smithi186 ceph-mon[67903]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-22T05:50:57.452 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:50:56 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:50:56] "GET /metrics HTTP/1.1" 200 38265 "" "Prometheus/2.43.0" 2024-03-22T05:50:58.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:57 smithi186 ceph-mon[67903]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:50:58.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:57 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-22T05:50:58.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:57 smithi005 ceph-mon[84705]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:50:58.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:57 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-22T05:50:58.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:57 smithi005 ceph-mon[89904]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:50:58.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:57 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-22T05:50:59.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:58 smithi186 ceph-mon[67903]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-22T05:50:59.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:58 smithi186 ceph-mon[67903]: Upgrade: osd.6 is safe to restart 2024-03-22T05:50:59.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:58 smithi005 ceph-mon[84705]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-22T05:50:59.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:58 smithi005 ceph-mon[84705]: Upgrade: osd.6 is safe to restart 2024-03-22T05:50:59.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:58 smithi005 ceph-mon[89904]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-22T05:50:59.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:58 smithi005 ceph-mon[89904]: Upgrade: osd.6 is safe to restart 2024-03-22T05:51:00.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:59 smithi186 ceph-mon[67903]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-22T05:51:00.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:59 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:00.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:59 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-22T05:51:00.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:50:59 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:51:00.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:59 smithi005 ceph-mon[84705]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-22T05:51:00.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:59 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:00.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:59 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-22T05:51:00.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:50:59 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:51:00.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:59 smithi005 ceph-mon[89904]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-22T05:51:00.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:59 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:00.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:59 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-22T05:51:00.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:50:59 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:51:01.251 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:00 smithi186 ceph-mon[67903]: Upgrade: Updating osd.6 2024-03-22T05:51:01.251 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:00 smithi186 ceph-mon[67903]: Deploying daemon osd.6 on smithi186 2024-03-22T05:51:01.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:00 smithi005 ceph-mon[84705]: Upgrade: Updating osd.6 2024-03-22T05:51:01.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:00 smithi005 ceph-mon[84705]: Deploying daemon osd.6 on smithi186 2024-03-22T05:51:01.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:00 smithi005 ceph-mon[89904]: Upgrade: Updating osd.6 2024-03-22T05:51:01.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:00 smithi005 ceph-mon[89904]: Deploying daemon osd.6 on smithi186 2024-03-22T05:51:02.284 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:01 smithi186 ceph-mon[67903]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-22T05:51:02.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:01 smithi005 ceph-mon[89904]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-22T05:51:02.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:01 smithi005 ceph-mon[84705]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-22T05:51:03.180 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:51:02 smithi186 systemd[1]: Stopping Ceph osd.6 for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:51:03.544 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:51:03 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6[34894]: 2024-03-22T05:51:03.180+0000 7f40a5676700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T05:51:03.544 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:51:03 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6[34894]: 2024-03-22T05:51:03.180+0000 7f40a5676700 -1 osd.6 135 *** Got signal Terminated *** 2024-03-22T05:51:03.544 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:51:03 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6[34894]: 2024-03-22T05:51:03.180+0000 7f40a5676700 -1 osd.6 135 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-22T05:51:04.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:03 smithi005 ceph-mon[84705]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:51:04.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:03 smithi005 ceph-mon[84705]: osd.6 marked itself down and dead 2024-03-22T05:51:04.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:03 smithi005 ceph-mon[89904]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:51:04.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:03 smithi005 ceph-mon[89904]: osd.6 marked itself down and dead 2024-03-22T05:51:04.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:03 smithi186 ceph-mon[67903]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:51:04.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:03 smithi186 ceph-mon[67903]: osd.6 marked itself down and dead 2024-03-22T05:51:04.646 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:51:04 smithi186 podman[86743]: 2024-03-22 05:51:04.361204743 +0000 UTC m=+1.307455646 container died 704ac49c68c4ad081b93f1e2e039080b24fbeed0dfa3b7a08a170487485c62ef (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6, version=8, name=centos-stream, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, vcs-type=git, io.buildah.version=1.19.8, release=754, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True, distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=) 2024-03-22T05:51:04.947 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:51:04 smithi186 podman[86743]: 2024-03-22 05:51:04.646957244 +0000 UTC m=+1.593208144 container cleanup 704ac49c68c4ad081b93f1e2e039080b24fbeed0dfa3b7a08a170487485c62ef (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, version=8, io.buildah.version=1.19.8, RELEASE=HEAD, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, name=centos-stream, distribution-scope=public, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, release=754, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=) 2024-03-22T05:51:04.947 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:51:04 smithi186 bash[86743]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6 2024-03-22T05:51:05.278 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:04 smithi186 ceph-mon[67903]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:51:05.278 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:04 smithi186 ceph-mon[67903]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-22T05:51:05.278 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:04 smithi186 ceph-mon[67903]: osdmap e136: 8 total, 7 up, 8 in 2024-03-22T05:51:05.278 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:51:04 smithi186 podman[86759]: 2024-03-22 05:51:04.94805781 +0000 UTC m=+0.587631351 container remove 704ac49c68c4ad081b93f1e2e039080b24fbeed0dfa3b7a08a170487485c62ef (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6, GIT_CLEAN=True, architecture=x86_64, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, version=8) 2024-03-22T05:51:05.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:04 smithi005 ceph-mon[89904]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:51:05.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:04 smithi005 ceph-mon[89904]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-22T05:51:05.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:04 smithi005 ceph-mon[89904]: osdmap e136: 8 total, 7 up, 8 in 2024-03-22T05:51:05.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:04 smithi005 ceph-mon[84705]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:51:05.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:04 smithi005 ceph-mon[84705]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-22T05:51:05.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:04 smithi005 ceph-mon[84705]: osdmap e136: 8 total, 7 up, 8 in 2024-03-22T05:51:05.543 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:51:05 smithi186 podman[86853]: 2024-03-22 05:51:05.181153561 +0000 UTC m=+0.018357167 image pull aa461e7c5e7f322b169c789b389d594b994ea7ffcbc9e24e563a3ec9b69bc3c1 quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9 2024-03-22T05:51:05.543 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:51:05 smithi186 podman[86853]: 2024-03-22 05:51:05.324271213 +0000 UTC m=+0.161474811 container create 29d314f5644569613a0a8aa950b6f359a89d8489a85b30eaca46d5d66db98f69 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, ceph=True, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD) 2024-03-22T05:51:05.794 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:51:05 smithi186 podman[86853]: 2024-03-22 05:51:05.543691416 +0000 UTC m=+0.380895024 container init 29d314f5644569613a0a8aa950b6f359a89d8489a85b30eaca46d5d66db98f69 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-22T05:51:05.794 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:51:05 smithi186 podman[86853]: 2024-03-22 05:51:05.547252323 +0000 UTC m=+0.384455932 container start 29d314f5644569613a0a8aa950b6f359a89d8489a85b30eaca46d5d66db98f69 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-22T05:51:05.794 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:51:05 smithi186 podman[86853]: 2024-03-22 05:51:05.626828037 +0000 UTC m=+0.464031652 container attach 29d314f5644569613a0a8aa950b6f359a89d8489a85b30eaca46d5d66db98f69 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6-deactivate, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-22T05:51:06.129 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:05 smithi186 ceph-mon[67903]: pgmap v245: 161 pgs: 4 peering, 13 stale+active+clean, 144 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:51:06.130 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:05 smithi186 ceph-mon[67903]: osdmap e137: 8 total, 7 up, 8 in 2024-03-22T05:51:06.130 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:51:05 smithi186 podman[86853]: 2024-03-22 05:51:05.844340635 +0000 UTC m=+0.681544225 container died 29d314f5644569613a0a8aa950b6f359a89d8489a85b30eaca46d5d66db98f69 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217) 2024-03-22T05:51:06.390 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:51:06 smithi186 podman[86879]: 2024-03-22 05:51:06.363712424 +0000 UTC m=+0.514620832 container remove 29d314f5644569613a0a8aa950b6f359a89d8489a85b30eaca46d5d66db98f69 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-22T05:51:06.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:05 smithi005 ceph-mon[84705]: pgmap v245: 161 pgs: 4 peering, 13 stale+active+clean, 144 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:51:06.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:05 smithi005 ceph-mon[84705]: osdmap e137: 8 total, 7 up, 8 in 2024-03-22T05:51:06.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:05 smithi005 ceph-mon[89904]: pgmap v245: 161 pgs: 4 peering, 13 stale+active+clean, 144 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:51:06.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:05 smithi005 ceph-mon[89904]: osdmap e137: 8 total, 7 up, 8 in 2024-03-22T05:51:06.794 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:51:06 smithi186 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@osd.6.service: Deactivated successfully. 2024-03-22T05:51:06.794 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:51:06 smithi186 systemd[1]: Stopped Ceph osd.6 for f59f36e2-e80a-11ee-95cd-87774f69a715. 2024-03-22T05:51:06.794 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:51:06 smithi186 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@osd.6.service: Consumed 14.721s CPU time. 2024-03-22T05:51:06.794 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:51:06 smithi186 systemd[1]: Starting Ceph osd.6 for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:51:07.067 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:07 smithi186 ceph-mon[67903]: Health check failed: Degraded data redundancy: 28/723 objects degraded (3.873%), 8 pgs degraded (PG_DEGRADED) 2024-03-22T05:51:07.067 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:51:06 smithi186 podman[86985]: 2024-03-22 05:51:06.707944711 +0000 UTC m=+0.017745055 image pull aa461e7c5e7f322b169c789b389d594b994ea7ffcbc9e24e563a3ec9b69bc3c1 quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9 2024-03-22T05:51:07.067 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:51:06 smithi186 podman[86985]: 2024-03-22 05:51:06.847993176 +0000 UTC m=+0.157793507 container create de48947b148f6254870a2af46ea92ee9f7db67e284c9223edb9bb91553656c75 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6-activate, org.label-schema.build-date=20240319, GIT_CLEAN=True, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-03-22T05:51:07.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:07 smithi005 ceph-mon[84705]: Health check failed: Degraded data redundancy: 28/723 objects degraded (3.873%), 8 pgs degraded (PG_DEGRADED) 2024-03-22T05:51:07.453 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:51:06] "GET /metrics HTTP/1.1" 200 38265 "" "Prometheus/2.43.0" 2024-03-22T05:51:07.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:07 smithi005 ceph-mon[89904]: Health check failed: Degraded data redundancy: 28/723 objects degraded (3.873%), 8 pgs degraded (PG_DEGRADED) 2024-03-22T05:51:07.515 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:51:07 smithi186 podman[86985]: 2024-03-22 05:51:07.067718248 +0000 UTC m=+0.377518588 container init de48947b148f6254870a2af46ea92ee9f7db67e284c9223edb9bb91553656c75 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6-activate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319) 2024-03-22T05:51:07.515 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:51:07 smithi186 podman[86985]: 2024-03-22 05:51:07.071306257 +0000 UTC m=+0.381106588 container start de48947b148f6254870a2af46ea92ee9f7db67e284c9223edb9bb91553656c75 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6-activate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0) 2024-03-22T05:51:07.515 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:51:07 smithi186 podman[86985]: 2024-03-22 05:51:07.14224936 +0000 UTC m=+0.452049705 container attach de48947b148f6254870a2af46ea92ee9f7db67e284c9223edb9bb91553656c75 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6-activate, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-22T05:51:07.794 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:51:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6-activate[87000]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-22T05:51:07.794 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:51:07 smithi186 bash[86985]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-22T05:51:07.794 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:51:07 smithi186 bash[86985]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-22T05:51:07.794 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:51:07 smithi186 bash[86985]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-22T05:51:07.795 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:51:07 smithi186 bash[86985]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-22T05:51:07.795 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:51:07 smithi186 bash[86985]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-03-22T05:51:07.795 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:51:07 smithi186 bash[86985]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-22T05:51:07.795 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:51:07 smithi186 bash[86985]: --> ceph-volume raw activate successful for osd ID: 6 2024-03-22T05:51:07.795 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:51:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6-activate[87000]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-22T05:51:07.795 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:51:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6-activate[87000]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-22T05:51:07.795 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:51:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6-activate[87000]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-22T05:51:07.795 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:51:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6-activate[87000]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-03-22T05:51:07.795 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:51:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6-activate[87000]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-22T05:51:07.795 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:51:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6-activate[87000]: --> ceph-volume raw activate successful for osd ID: 6 2024-03-22T05:51:07.795 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:51:07 smithi186 podman[86985]: 2024-03-22 05:51:07.541224663 +0000 UTC m=+0.851024989 container died de48947b148f6254870a2af46ea92ee9f7db67e284c9223edb9bb91553656c75 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-22T05:51:08.128 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:08 smithi186 ceph-mon[67903]: pgmap v247: 161 pgs: 8 active+undersized+degraded, 4 peering, 8 stale+active+clean, 15 active+undersized, 126 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s; 28/723 objects degraded (3.873%) 2024-03-22T05:51:08.128 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:08 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:08.129 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:08 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:51:08.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:08 smithi005 ceph-mon[89904]: pgmap v247: 161 pgs: 8 active+undersized+degraded, 4 peering, 8 stale+active+clean, 15 active+undersized, 126 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s; 28/723 objects degraded (3.873%) 2024-03-22T05:51:08.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:08 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:08.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:08 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:51:08.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:08 smithi005 ceph-mon[84705]: pgmap v247: 161 pgs: 8 active+undersized+degraded, 4 peering, 8 stale+active+clean, 15 active+undersized, 126 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s; 28/723 objects degraded (3.873%) 2024-03-22T05:51:08.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:08 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:08.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:08 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:51:08.472 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:51:08 smithi186 podman[87132]: 2024-03-22 05:51:08.12932444 +0000 UTC m=+0.582770909 container remove de48947b148f6254870a2af46ea92ee9f7db67e284c9223edb9bb91553656c75 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6-activate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS) 2024-03-22T05:51:08.472 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:51:08 smithi186 podman[87197]: 2024-03-22 05:51:08.246801964 +0000 UTC m=+0.018683816 image pull aa461e7c5e7f322b169c789b389d594b994ea7ffcbc9e24e563a3ec9b69bc3c1 quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9 2024-03-22T05:51:08.472 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:51:08 smithi186 podman[87197]: 2024-03-22 05:51:08.355051997 +0000 UTC m=+0.126933831 container create db7c81962c65d77d151a0f80f2a64c13f9bc914764962ce08157992681ee2cac (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-22T05:51:09.044 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:51:08 smithi186 podman[87197]: 2024-03-22 05:51:08.583097804 +0000 UTC m=+0.354979651 container init db7c81962c65d77d151a0f80f2a64c13f9bc914764962ce08157992681ee2cac (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-22T05:51:09.044 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:51:08 smithi186 podman[87197]: 2024-03-22 05:51:08.586418203 +0000 UTC m=+0.358300055 container start db7c81962c65d77d151a0f80f2a64c13f9bc914764962ce08157992681ee2cac (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-22T05:51:09.044 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:51:08 smithi186 ceph-osd[87216]: -- 172.21.15.186:0/1592993254 <== mon.2 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f53539d680 con 0x55f535366c00 2024-03-22T05:51:09.044 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:51:08 smithi186 bash[87197]: db7c81962c65d77d151a0f80f2a64c13f9bc914764962ce08157992681ee2cac 2024-03-22T05:51:09.044 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:51:08 smithi186 systemd[1]: Started Ceph osd.6 for f59f36e2-e80a-11ee-95cd-87774f69a715. 2024-03-22T05:51:09.487 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:51:09 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6[87212]: 2024-03-22T05:51:09.129+0000 7f14221947c0 -1 Falling back to public interface 2024-03-22T05:51:09.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:09 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:09.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:09 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:09.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:09 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:51:09.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:09 smithi186 ceph-mon[67903]: pgmap v248: 161 pgs: 15 active+undersized+degraded, 4 peering, 34 active+undersized, 108 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s; 71/723 objects degraded (9.820%) 2024-03-22T05:51:10.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:09 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:10.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:09 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:10.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:09 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:51:10.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:09 smithi005 ceph-mon[84705]: pgmap v248: 161 pgs: 15 active+undersized+degraded, 4 peering, 34 active+undersized, 108 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s; 71/723 objects degraded (9.820%) 2024-03-22T05:51:10.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:09 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:10.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:09 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:10.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:09 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:51:10.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:09 smithi005 ceph-mon[89904]: pgmap v248: 161 pgs: 15 active+undersized+degraded, 4 peering, 34 active+undersized, 108 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s; 71/723 objects degraded (9.820%) 2024-03-22T05:51:11.983 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:51:11 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6[87212]: 2024-03-22T05:51:11.826+0000 7f14221947c0 -1 osd.6 0 read_superblock omap replica is missing. 2024-03-22T05:51:11.983 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:51:11 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6[87212]: 2024-03-22T05:51:11.852+0000 7f14221947c0 -1 osd.6 135 log_to_monitors true 2024-03-22T05:51:12.248 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:11 smithi005 ceph-mon[84705]: pgmap v249: 161 pgs: 17 active+undersized+degraded, 36 active+undersized, 108 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 83/723 objects degraded (11.480%) 2024-03-22T05:51:12.248 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:11 smithi005 ceph-mon[84705]: from='osd.6 [v2:172.21.15.186:6816/1404340816,v1:172.21.15.186:6817/1404340816]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-22T05:51:12.248 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:11 smithi005 ceph-mon[84705]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-22T05:51:12.249 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:11 smithi005 ceph-mon[89904]: pgmap v249: 161 pgs: 17 active+undersized+degraded, 36 active+undersized, 108 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 83/723 objects degraded (11.480%) 2024-03-22T05:51:12.249 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:11 smithi005 ceph-mon[89904]: from='osd.6 [v2:172.21.15.186:6816/1404340816,v1:172.21.15.186:6817/1404340816]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-22T05:51:12.249 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:11 smithi005 ceph-mon[89904]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-22T05:51:12.263 INFO:teuthology.orchestra.run.smithi005.stdout:true 2024-03-22T05:51:12.277 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:11 smithi186 ceph-mon[67903]: pgmap v249: 161 pgs: 17 active+undersized+degraded, 36 active+undersized, 108 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 83/723 objects degraded (11.480%) 2024-03-22T05:51:12.277 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:11 smithi186 ceph-mon[67903]: from='osd.6 [v2:172.21.15.186:6816/1404340816,v1:172.21.15.186:6817/1404340816]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-22T05:51:12.277 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:11 smithi186 ceph-mon[67903]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-22T05:51:12.543 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:51:12 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6[87212]: 2024-03-22T05:51:12.449+0000 7f140f5e4700 -1 osd.6 135 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-22T05:51:12.979 INFO:teuthology.orchestra.run.smithi005.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T05:51:12.979 INFO:teuthology.orchestra.run.smithi005.stdout:alertmanager.a smithi005 *:9093,9094 running (8m) 110s ago 30m 28.0M - 0.25.0 c8568f914cd2 ee99dfc435e8 2024-03-22T05:51:12.979 INFO:teuthology.orchestra.run.smithi005.stdout:grafana.a smithi186 *:3000 running (8m) 0s ago 30m 81.2M - 9.4.12 2bacad6d85d8 16fd57dbbd71 2024-03-22T05:51:12.979 INFO:teuthology.orchestra.run.smithi005.stdout:iscsi.foo.smithi005.mcdwbp smithi005 unknown 110s ago 29m 64.6M - 3.5 e1d6a67b021e 5215e03bf81b 2024-03-22T05:51:12.979 INFO:teuthology.orchestra.run.smithi005.stdout:mgr.x smithi186 *:8443,9283,8765 running (9m) 0s ago 35m 453M - 19.0.0-2299-gea01217f aa461e7c5e7f c556421a73f3 2024-03-22T05:51:12.979 INFO:teuthology.orchestra.run.smithi005.stdout:mgr.y smithi005 *:8443,9283,8765 running (19m) 110s ago 36m 511M - 19.0.0-2299-gea01217f aa461e7c5e7f 62a813eebc0d 2024-03-22T05:51:12.979 INFO:teuthology.orchestra.run.smithi005.stdout:mon.a smithi005 running (6m) 110s ago 36m 55.3M 2048M 19.0.0-2299-gea01217f aa461e7c5e7f 111a2aa24dd9 2024-03-22T05:51:12.979 INFO:teuthology.orchestra.run.smithi005.stdout:mon.b smithi186 running (7m) 0s ago 35m 53.0M 2048M 19.0.0-2299-gea01217f aa461e7c5e7f c1798feac15b 2024-03-22T05:51:12.979 INFO:teuthology.orchestra.run.smithi005.stdout:mon.c smithi005 running (6m) 110s ago 35m 42.0M 2048M 19.0.0-2299-gea01217f aa461e7c5e7f 708c72ff74f5 2024-03-22T05:51:12.979 INFO:teuthology.orchestra.run.smithi005.stdout:node-exporter.a smithi005 *:9100 running (9m) 110s ago 31m 16.8M - 1.5.0 0da6a335fe13 f635cd72b9d3 2024-03-22T05:51:12.980 INFO:teuthology.orchestra.run.smithi005.stdout:node-exporter.b smithi186 *:9100 running (9m) 0s ago 31m 15.2M - 1.5.0 0da6a335fe13 d7e0b3d20191 2024-03-22T05:51:12.980 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0 smithi005 running (4m) 110s ago 34m 44.8M 2993M 19.0.0-2299-gea01217f aa461e7c5e7f 31b6cdee5f5b 2024-03-22T05:51:12.980 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1 smithi005 running (3m) 110s ago 34m 44.2M 2993M 19.0.0-2299-gea01217f aa461e7c5e7f 7d21bc9fd3c2 2024-03-22T05:51:12.980 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2 smithi005 running (2m) 110s ago 34m 42.0M 2993M 19.0.0-2299-gea01217f aa461e7c5e7f 628204cb7c98 2024-03-22T05:51:12.980 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3 smithi005 running (115s) 110s ago 33m 14.9M 2993M 19.0.0-2299-gea01217f aa461e7c5e7f d31a01a0230c 2024-03-22T05:51:12.980 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4 smithi186 running (78s) 0s ago 33m 54.6M 3248M 19.0.0-2299-gea01217f aa461e7c5e7f 8f7619b726f3 2024-03-22T05:51:12.980 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5 smithi186 running (41s) 0s ago 32m 49.9M 3248M 19.0.0-2299-gea01217f aa461e7c5e7f b80f7eceb2d7 2024-03-22T05:51:12.980 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6 smithi186 running (4s) 0s ago 32m 12.7M 3248M 19.0.0-2299-gea01217f aa461e7c5e7f db7c81962c65 2024-03-22T05:51:12.980 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7 smithi186 running (32m) 0s ago 32m 60.6M 3248M 17.2.0 e1d6a67b021e f415b90edf63 2024-03-22T05:51:12.980 INFO:teuthology.orchestra.run.smithi005.stdout:prometheus.a smithi186 *:9095 running (9m) 0s ago 31m 72.0M - 2.43.0 a07b618ecd1d a5a043588d6d 2024-03-22T05:51:12.980 INFO:teuthology.orchestra.run.smithi005.stdout:rgw.foo.smithi005.exdhqu smithi005 *:8000 running (29m) 110s ago 29m 81.5M - 17.2.0 e1d6a67b021e 6bb27fe11553 2024-03-22T05:51:12.981 INFO:teuthology.orchestra.run.smithi005.stdout:rgw.foo.smithi186.mcgodk smithi186 *:8000 running (29m) 0s ago 29m 80.1M - 17.2.0 e1d6a67b021e 19b5c20a34f7 2024-03-22T05:51:13.257 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:12 smithi005 ceph-mon[84705]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-22T05:51:13.257 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:12 smithi005 ceph-mon[84705]: osdmap e138: 8 total, 7 up, 8 in 2024-03-22T05:51:13.257 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:12 smithi005 ceph-mon[84705]: from='osd.6 [v2:172.21.15.186:6816/1404340816,v1:172.21.15.186:6817/1404340816]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-03-22T05:51:13.258 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:12 smithi005 ceph-mon[84705]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-03-22T05:51:13.258 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:12 smithi005 ceph-mon[84705]: from='client.64513 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:51:13.258 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:12 smithi005 ceph-mon[84705]: Health check update: Degraded data redundancy: 83/723 objects degraded (11.480%), 17 pgs degraded (PG_DEGRADED) 2024-03-22T05:51:13.258 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:12 smithi005 ceph-mon[84705]: from='osd.6 ' entity='osd.6' 2024-03-22T05:51:13.258 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:12 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:13.258 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:12 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:13.258 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:12 smithi005 ceph-mon[89904]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-22T05:51:13.258 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:12 smithi005 ceph-mon[89904]: osdmap e138: 8 total, 7 up, 8 in 2024-03-22T05:51:13.258 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:12 smithi005 ceph-mon[89904]: from='osd.6 [v2:172.21.15.186:6816/1404340816,v1:172.21.15.186:6817/1404340816]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-03-22T05:51:13.258 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:12 smithi005 ceph-mon[89904]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-03-22T05:51:13.258 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:12 smithi005 ceph-mon[89904]: from='client.64513 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:51:13.258 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:12 smithi005 ceph-mon[89904]: Health check update: Degraded data redundancy: 83/723 objects degraded (11.480%), 17 pgs degraded (PG_DEGRADED) 2024-03-22T05:51:13.258 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:12 smithi005 ceph-mon[89904]: from='osd.6 ' entity='osd.6' 2024-03-22T05:51:13.259 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:12 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:13.259 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:12 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:13.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:12 smithi186 ceph-mon[67903]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-22T05:51:13.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:12 smithi186 ceph-mon[67903]: osdmap e138: 8 total, 7 up, 8 in 2024-03-22T05:51:13.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:12 smithi186 ceph-mon[67903]: from='osd.6 [v2:172.21.15.186:6816/1404340816,v1:172.21.15.186:6817/1404340816]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-03-22T05:51:13.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:12 smithi186 ceph-mon[67903]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-03-22T05:51:13.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:12 smithi186 ceph-mon[67903]: from='client.64513 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:51:13.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:12 smithi186 ceph-mon[67903]: Health check update: Degraded data redundancy: 83/723 objects degraded (11.480%), 17 pgs degraded (PG_DEGRADED) 2024-03-22T05:51:13.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:12 smithi186 ceph-mon[67903]: from='osd.6 ' entity='osd.6' 2024-03-22T05:51:13.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:12 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:13.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:12 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:13.434 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-03-22T05:51:13.435 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": { 2024-03-22T05:51:13.435 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev)": 3 2024-03-22T05:51:13.435 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:51:13.435 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr": { 2024-03-22T05:51:13.435 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev)": 2 2024-03-22T05:51:13.435 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:51:13.435 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": { 2024-03-22T05:51:13.435 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-03-22T05:51:13.435 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev)": 6 2024-03-22T05:51:13.435 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:51:13.435 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw": { 2024-03-22T05:51:13.435 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-22T05:51:13.435 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:51:13.436 INFO:teuthology.orchestra.run.smithi005.stdout: "overall": { 2024-03-22T05:51:13.436 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3, 2024-03-22T05:51:13.436 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev)": 11 2024-03-22T05:51:13.436 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-03-22T05:51:13.436 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-03-22T05:51:13.798 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-03-22T05:51:13.798 INFO:teuthology.orchestra.run.smithi005.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9", 2024-03-22T05:51:13.798 INFO:teuthology.orchestra.run.smithi005.stdout: "in_progress": true, 2024-03-22T05:51:13.798 INFO:teuthology.orchestra.run.smithi005.stdout: "which": "Upgrading daemons of type(s) crash,osd", 2024-03-22T05:51:13.798 INFO:teuthology.orchestra.run.smithi005.stdout: "services_complete": [], 2024-03-22T05:51:13.798 INFO:teuthology.orchestra.run.smithi005.stdout: "progress": "7/8 daemons upgraded", 2024-03-22T05:51:13.798 INFO:teuthology.orchestra.run.smithi005.stdout: "message": "Currently upgrading osd daemons", 2024-03-22T05:51:13.798 INFO:teuthology.orchestra.run.smithi005.stdout: "is_paused": false 2024-03-22T05:51:13.798 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-03-22T05:51:14.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:13 smithi005 ceph-mon[84705]: from='client.64519 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:51:14.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:13 smithi005 ceph-mon[84705]: pgmap v251: 161 pgs: 17 active+undersized+degraded, 36 active+undersized, 108 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s; 83/723 objects degraded (11.480%) 2024-03-22T05:51:14.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:13 smithi005 ceph-mon[84705]: from='client.64525 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:51:14.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:13 smithi005 ceph-mon[84705]: from='client.? 172.21.15.5:0/1600349634' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:51:14.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:13 smithi005 ceph-mon[84705]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-22T05:51:14.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:13 smithi005 ceph-mon[84705]: osd.6 [v2:172.21.15.186:6816/1404340816,v1:172.21.15.186:6817/1404340816] boot 2024-03-22T05:51:14.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:13 smithi005 ceph-mon[84705]: osdmap e139: 8 total, 8 up, 8 in 2024-03-22T05:51:14.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:13 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-22T05:51:14.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:13 smithi005 ceph-mon[89904]: from='client.64519 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:51:14.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:13 smithi005 ceph-mon[89904]: pgmap v251: 161 pgs: 17 active+undersized+degraded, 36 active+undersized, 108 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s; 83/723 objects degraded (11.480%) 2024-03-22T05:51:14.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:13 smithi005 ceph-mon[89904]: from='client.64525 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:51:14.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:13 smithi005 ceph-mon[89904]: from='client.? 172.21.15.5:0/1600349634' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:51:14.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:13 smithi005 ceph-mon[89904]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-22T05:51:14.204 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:13 smithi005 ceph-mon[89904]: osd.6 [v2:172.21.15.186:6816/1404340816,v1:172.21.15.186:6817/1404340816] boot 2024-03-22T05:51:14.204 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:13 smithi005 ceph-mon[89904]: osdmap e139: 8 total, 8 up, 8 in 2024-03-22T05:51:14.204 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:13 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-22T05:51:14.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:13 smithi186 ceph-mon[67903]: from='client.64519 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:51:14.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:13 smithi186 ceph-mon[67903]: pgmap v251: 161 pgs: 17 active+undersized+degraded, 36 active+undersized, 108 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s; 83/723 objects degraded (11.480%) 2024-03-22T05:51:14.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:13 smithi186 ceph-mon[67903]: from='client.64525 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:51:14.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:13 smithi186 ceph-mon[67903]: from='client.? 172.21.15.5:0/1600349634' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:51:14.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:13 smithi186 ceph-mon[67903]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-22T05:51:14.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:13 smithi186 ceph-mon[67903]: osd.6 [v2:172.21.15.186:6816/1404340816,v1:172.21.15.186:6817/1404340816] boot 2024-03-22T05:51:14.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:13 smithi186 ceph-mon[67903]: osdmap e139: 8 total, 8 up, 8 in 2024-03-22T05:51:14.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:13 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-22T05:51:15.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:15 smithi186 ceph-mon[67903]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:51:15.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:15 smithi186 ceph-mon[67903]: from='client.54603 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:51:15.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:15 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:15.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:15 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:15.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:15 smithi186 ceph-mon[67903]: osdmap e140: 8 total, 8 up, 8 in 2024-03-22T05:51:15.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:15 smithi005 ceph-mon[84705]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:51:15.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:15 smithi005 ceph-mon[84705]: from='client.54603 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:51:15.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:15 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:15.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:15 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:15.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:15 smithi005 ceph-mon[84705]: osdmap e140: 8 total, 8 up, 8 in 2024-03-22T05:51:15.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:15 smithi005 ceph-mon[89904]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:51:15.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:15 smithi005 ceph-mon[89904]: from='client.54603 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:51:15.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:15 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:15.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:15 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:15.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:15 smithi005 ceph-mon[89904]: osdmap e140: 8 total, 8 up, 8 in 2024-03-22T05:51:16.293 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:16 smithi186 ceph-mon[67903]: pgmap v254: 161 pgs: 13 peering, 12 active+undersized+degraded, 28 active+undersized, 108 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 55/723 objects degraded (7.607%) 2024-03-22T05:51:16.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:16 smithi005 ceph-mon[84705]: pgmap v254: 161 pgs: 13 peering, 12 active+undersized+degraded, 28 active+undersized, 108 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 55/723 objects degraded (7.607%) 2024-03-22T05:51:16.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:16 smithi005 ceph-mon[89904]: pgmap v254: 161 pgs: 13 peering, 12 active+undersized+degraded, 28 active+undersized, 108 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 55/723 objects degraded (7.607%) 2024-03-22T05:51:17.452 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:16 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:51:16] "GET /metrics HTTP/1.1" 200 38261 "" "Prometheus/2.43.0" 2024-03-22T05:51:18.379 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:18 smithi005 ceph-mon[84705]: pgmap v255: 161 pgs: 13 peering, 7 active+undersized+degraded, 15 active+undersized, 126 active+clean; 585 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 58 op/s; 31/723 objects degraded (4.288%) 2024-03-22T05:51:18.379 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:18 smithi005 ceph-mon[84705]: Health check update: Degraded data redundancy: 31/723 objects degraded (4.288%), 7 pgs degraded (PG_DEGRADED) 2024-03-22T05:51:18.379 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:18 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:18.379 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:18 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:18.379 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:18 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:51:18.379 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:18 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:51:18.380 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:18 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:18.380 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:18 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:51:18.380 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:18 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:51:18.380 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:18 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:51:18.380 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:18 smithi005 ceph-mon[89904]: pgmap v255: 161 pgs: 13 peering, 7 active+undersized+degraded, 15 active+undersized, 126 active+clean; 585 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 58 op/s; 31/723 objects degraded (4.288%) 2024-03-22T05:51:18.380 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:18 smithi005 ceph-mon[89904]: Health check update: Degraded data redundancy: 31/723 objects degraded (4.288%), 7 pgs degraded (PG_DEGRADED) 2024-03-22T05:51:18.380 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:18 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:18.380 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:18 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:18.380 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:18 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:51:18.380 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:18 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:51:18.380 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:18 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:18.380 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:18 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:51:18.380 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:18 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:51:18.381 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:18 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:51:18.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:18 smithi186 ceph-mon[67903]: pgmap v255: 161 pgs: 13 peering, 7 active+undersized+degraded, 15 active+undersized, 126 active+clean; 585 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 58 op/s; 31/723 objects degraded (4.288%) 2024-03-22T05:51:18.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:18 smithi186 ceph-mon[67903]: Health check update: Degraded data redundancy: 31/723 objects degraded (4.288%), 7 pgs degraded (PG_DEGRADED) 2024-03-22T05:51:18.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:18 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:18.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:18 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:18.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:18 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:51:18.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:18 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:51:18.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:18 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:18.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:18 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:51:18.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:18 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:51:18.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:18 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:51:19.414 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:19 smithi005 ceph-mon[84705]: pgmap v256: 161 pgs: 13 peering, 3 active+undersized+degraded, 6 active+undersized, 139 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 69 op/s; 17/723 objects degraded (2.351%) 2024-03-22T05:51:19.414 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:19 smithi005 ceph-mon[84705]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:51:19.414 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:19 smithi005 ceph-mon[84705]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:51:19.414 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:19 smithi005 ceph-mon[89904]: pgmap v256: 161 pgs: 13 peering, 3 active+undersized+degraded, 6 active+undersized, 139 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 69 op/s; 17/723 objects degraded (2.351%) 2024-03-22T05:51:19.414 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:19 smithi005 ceph-mon[89904]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:51:19.415 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:19 smithi005 ceph-mon[89904]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:51:19.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:19 smithi186 ceph-mon[67903]: pgmap v256: 161 pgs: 13 peering, 3 active+undersized+degraded, 6 active+undersized, 139 active+clean; 585 KiB data, 237 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 69 op/s; 17/723 objects degraded (2.351%) 2024-03-22T05:51:19.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:19 smithi186 ceph-mon[67903]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:51:19.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:19 smithi186 ceph-mon[67903]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:51:19.702 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:51:19.627+0000 7f7f0ecc9700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:51:19.703 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:51:19.703 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:51:19.703 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:51:19.703 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:51:19.703 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:51:19.703 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:51:19.703 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:51:19.703 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:51:19.703 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:51:19.703 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:51:19.703 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:51:19.703 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:51:19.703 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:51:19.704 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:51:19.704 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:51:19.704 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:51:19.704 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:51:19.704 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:51:19.704 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:51:19.704 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:51:19.704 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:51:19.704 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:51:19.705 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:51:19.705 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:51:19.705 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:51:19.705 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:51:19.705 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:51:19.705 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:51:19.705 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return future.result(timeout) 2024-03-22T05:51:19.705 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:51:19.705 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.__get_result() 2024-03-22T05:51:19.705 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:51:19.705 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: raise self._exception 2024-03-22T05:51:19.705 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:51:19.705 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: config_blobs=daemon_spec.final_config, 2024-03-22T05:51:19.705 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:51:19.705 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:51:19.705 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:51:19.706 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:51:19.706 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:51:19.706 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:51:19.706 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:51:19.706 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:51:19.706 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:51:19.706 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:51:19.706 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:51:19.706 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:51:19.706 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:51:19.706 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:51:19.706 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:51:19.706 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:51:19.706 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:51:19.706 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:51:19.706 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:51:19.707 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:51:19.707 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:51:19.707 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:51:19.707 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:51:20.397 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:20 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:51:20.397 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:20 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:51:20.397 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:20 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:51:20.397 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:20 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:51:20.397 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:20 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-22T05:51:20.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:20 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:51:20.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:20 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:51:20.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:20 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:51:20.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:20 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:51:20.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:20 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-22T05:51:20.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:20 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:51:20.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:20 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:51:20.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:20 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:51:20.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:20 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:51:20.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:20 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-22T05:51:21.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:21 smithi005 ceph-mon[89904]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 105 op/s 2024-03-22T05:51:21.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:21 smithi005 ceph-mon[89904]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:51:21.453 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:51:21.453 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:51:21.453 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:51:21.453 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:51:21.453 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:51:21.453 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:51:21.453 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:51:21.453 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:51:21.453 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:51:21.453 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:51:21.453 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:51:21.454 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:51:21.454 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:51:21.454 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:51:21.454 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:51:21.454 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:51:21.454 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:51:21.454 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:51:21.454 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:51:21.454 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:51:21.454 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:51:21.454 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:51:21.454 INFO:journalctl@ceph.mon.c.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:51:21.454 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:51:21.454 INFO:journalctl@ceph.mon.c.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:51:21.454 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:51:21.455 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:51:21.455 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:51:21.455 INFO:journalctl@ceph.mon.c.smithi005.stdout: return future.result(timeout) 2024-03-22T05:51:21.455 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:51:21.455 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.__get_result() 2024-03-22T05:51:21.455 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:51:21.455 INFO:journalctl@ceph.mon.c.smithi005.stdout: raise self._exception 2024-03-22T05:51:21.455 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:51:21.455 INFO:journalctl@ceph.mon.c.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:51:21.455 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:51:21.455 INFO:journalctl@ceph.mon.c.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:51:21.455 INFO:journalctl@ceph.mon.c.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:51:21.455 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:51:21.455 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:51:21.455 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:51:21.456 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:51:21.456 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:51:21.456 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:51:21.456 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:51:21.456 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:51:21.456 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:51:21.456 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:51:21.456 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:51:21.456 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:51:21.456 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:51:21.456 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:51:21.456 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:51:21.456 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:21 smithi005 ceph-mon[84705]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 105 op/s 2024-03-22T05:51:21.457 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:21 smithi005 ceph-mon[84705]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:51:21.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:51:21.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:51:21.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:51:21.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:51:21.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:51:21.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:51:21.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:51:21.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:51:21.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:51:21.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:51:21.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:51:21.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:51:21.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:51:21.458 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:51:21.458 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:51:21.458 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:51:21.458 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:51:21.458 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:51:21.458 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:51:21.458 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:51:21.458 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:51:21.458 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:51:21.458 INFO:journalctl@ceph.mon.a.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:51:21.458 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:51:21.458 INFO:journalctl@ceph.mon.a.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:51:21.458 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:51:21.458 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:51:21.458 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:51:21.459 INFO:journalctl@ceph.mon.a.smithi005.stdout: return future.result(timeout) 2024-03-22T05:51:21.459 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:51:21.459 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.__get_result() 2024-03-22T05:51:21.459 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:51:21.459 INFO:journalctl@ceph.mon.a.smithi005.stdout: raise self._exception 2024-03-22T05:51:21.459 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:51:21.459 INFO:journalctl@ceph.mon.a.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:51:21.459 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:51:21.459 INFO:journalctl@ceph.mon.a.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:51:21.459 INFO:journalctl@ceph.mon.a.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:51:21.459 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:51:21.459 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:51:21.459 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:51:21.459 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:51:21.460 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:51:21.460 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:51:21.460 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:51:21.460 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:51:21.460 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:51:21.460 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:51:21.460 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:51:21.460 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:51:21.460 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:51:21.460 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:51:21.460 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:51:21.460 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:51:21.460 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:51:21.460 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:51:21.460 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:51:21.461 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:51:21.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:21 smithi005 ceph-mon[84705]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-22T05:51:21.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:21 smithi005 ceph-mon[84705]: Upgrade: osd.7 is safe to restart 2024-03-22T05:51:21.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:21 smithi005 ceph-mon[84705]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 17/723 objects degraded (2.351%), 3 pgs degraded) 2024-03-22T05:51:21.461 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:51:21.461 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:51:21.461 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:51:21.461 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:51:21.461 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:51:21.461 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:21 smithi005 ceph-mon[89904]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-22T05:51:21.461 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:21 smithi005 ceph-mon[89904]: Upgrade: osd.7 is safe to restart 2024-03-22T05:51:21.462 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:21 smithi005 ceph-mon[89904]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 17/723 objects degraded (2.351%), 3 pgs degraded) 2024-03-22T05:51:21.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:21 smithi186 ceph-mon[67903]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 105 op/s 2024-03-22T05:51:21.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:21 smithi186 ceph-mon[67903]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:51:21.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:51:21.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:51:21.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:51:21.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:51:21.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:51:21.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:51:21.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:51:21.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:51:21.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:51:21.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:51:21.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:51:21.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:51:21.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:51:21.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:51:21.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:51:21.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:51:21.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:51:21.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:51:21.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:51:21.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:51:21.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:51:21.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:51:21.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:51:21.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:51:21.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:51:21.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:51:21.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:51:21.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:51:21.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: return future.result(timeout) 2024-03-22T05:51:21.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:51:21.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.__get_result() 2024-03-22T05:51:21.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:51:21.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: raise self._exception 2024-03-22T05:51:21.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:51:21.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:51:21.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:51:21.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:51:21.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:51:21.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:51:21.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:51:21.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:51:21.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:51:21.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:51:21.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:51:21.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:51:21.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:51:21.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:51:21.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:51:21.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:51:21.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:51:21.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:51:21.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:51:21.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:51:21.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:51:21.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:51:21.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:51:21.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:51:21.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:51:21.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:21 smithi186 ceph-mon[67903]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-22T05:51:21.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:21 smithi186 ceph-mon[67903]: Upgrade: osd.7 is safe to restart 2024-03-22T05:51:21.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:21 smithi186 ceph-mon[67903]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 17/723 objects degraded (2.351%), 3 pgs degraded) 2024-03-22T05:51:22.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:22 smithi186 ceph-mon[67903]: Upgrade: Updating osd.7 2024-03-22T05:51:22.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:22 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:22.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:22 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-22T05:51:22.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:22 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:51:22.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:22 smithi186 ceph-mon[67903]: Deploying daemon osd.7 on smithi186 2024-03-22T05:51:22.684 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:22 smithi005 ceph-mon[84705]: Upgrade: Updating osd.7 2024-03-22T05:51:22.684 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:22 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:22.684 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:22 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-22T05:51:22.685 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:22 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:51:22.685 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:22 smithi005 ceph-mon[84705]: Deploying daemon osd.7 on smithi186 2024-03-22T05:51:22.685 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:22 smithi005 ceph-mon[89904]: Upgrade: Updating osd.7 2024-03-22T05:51:22.685 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:22 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:22.685 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:22 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-22T05:51:22.685 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:22 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:51:22.685 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:22 smithi005 ceph-mon[89904]: Deploying daemon osd.7 on smithi186 2024-03-22T05:51:23.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:23 smithi186 ceph-mon[67903]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-03-22T05:51:23.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:23 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:23.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:23 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:51:23.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:23 smithi005 ceph-mon[84705]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-03-22T05:51:23.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:23 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:23.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:23 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:51:23.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:23 smithi005 ceph-mon[89904]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 117 op/s 2024-03-22T05:51:23.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:23 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:23.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:23 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:51:24.794 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:51:24 smithi186 systemd[1]: Stopping Ceph osd.7 for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:51:24.794 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:51:24 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[37870]: 2024-03-22T05:51:24.770+0000 7f3960df8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T05:51:24.794 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:51:24 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[37870]: 2024-03-22T05:51:24.770+0000 7f3960df8700 -1 osd.7 140 *** Got signal Terminated *** 2024-03-22T05:51:24.794 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:51:24 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[37870]: 2024-03-22T05:51:24.770+0000 7f3960df8700 -1 osd.7 140 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-22T05:51:25.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:25 smithi005 ceph-mon[84705]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 89 KiB/s rd, 0 B/s wr, 147 op/s 2024-03-22T05:51:25.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:25 smithi005 ceph-mon[84705]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:51:25.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:25 smithi005 ceph-mon[84705]: osd.7 marked itself down and dead 2024-03-22T05:51:25.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:25 smithi005 ceph-mon[89904]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 89 KiB/s rd, 0 B/s wr, 147 op/s 2024-03-22T05:51:25.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:25 smithi005 ceph-mon[89904]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:51:25.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:25 smithi005 ceph-mon[89904]: osd.7 marked itself down and dead 2024-03-22T05:51:25.773 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:25 smithi186 ceph-mon[67903]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 89 KiB/s rd, 0 B/s wr, 147 op/s 2024-03-22T05:51:25.773 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:25 smithi186 ceph-mon[67903]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:51:25.774 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:25 smithi186 ceph-mon[67903]: osd.7 marked itself down and dead 2024-03-22T05:51:26.044 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:51:25 smithi186 podman[91698]: 2024-03-22 05:51:25.774408443 +0000 UTC m=+1.104222467 container died f415b90edf631586bbd43afe34b2c648ec61f559c337ff30e510e78a73714a1c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7, io.openshift.expose-services=, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, com.redhat.component=centos-stream-container, ceph=True, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, release=754, vcs-type=git, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, GIT_CLEAN=True, distribution-scope=public, version=8, vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870) 2024-03-22T05:51:26.340 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:51:26 smithi186 podman[91698]: 2024-03-22 05:51:26.17763835 +0000 UTC m=+1.507452381 container cleanup f415b90edf631586bbd43afe34b2c648ec61f559c337ff30e510e78a73714a1c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, GIT_CLEAN=True, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, ceph=True, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, RELEASE=HEAD, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8) 2024-03-22T05:51:26.341 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:51:26 smithi186 bash[91698]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7 2024-03-22T05:51:26.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:26 smithi005 ceph-mon[84705]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-22T05:51:26.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:26 smithi005 ceph-mon[84705]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-03-22T05:51:26.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:26 smithi005 ceph-mon[84705]: osdmap e141: 8 total, 7 up, 8 in 2024-03-22T05:51:26.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:26 smithi005 ceph-mon[84705]: osdmap e142: 8 total, 7 up, 8 in 2024-03-22T05:51:26.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:26 smithi005 ceph-mon[89904]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-22T05:51:26.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:26 smithi005 ceph-mon[89904]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-03-22T05:51:26.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:26 smithi005 ceph-mon[89904]: osdmap e141: 8 total, 7 up, 8 in 2024-03-22T05:51:26.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:26 smithi005 ceph-mon[89904]: osdmap e142: 8 total, 7 up, 8 in 2024-03-22T05:51:26.757 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:26 smithi186 ceph-mon[67903]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-22T05:51:26.757 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:26 smithi186 ceph-mon[67903]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-03-22T05:51:26.757 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:26 smithi186 ceph-mon[67903]: osdmap e141: 8 total, 7 up, 8 in 2024-03-22T05:51:26.757 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:26 smithi186 ceph-mon[67903]: osdmap e142: 8 total, 7 up, 8 in 2024-03-22T05:51:26.758 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:51:26 smithi186 podman[91715]: 2024-03-22 05:51:26.420831974 +0000 UTC m=+0.638211947 container remove f415b90edf631586bbd43afe34b2c648ec61f559c337ff30e510e78a73714a1c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, io.openshift.expose-services=, vcs-type=git, GIT_BRANCH=HEAD, name=centos-stream, ceph=True, vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8) 2024-03-22T05:51:27.044 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:51:26 smithi186 podman[91806]: 2024-03-22 05:51:26.661061384 +0000 UTC m=+0.018922908 image pull aa461e7c5e7f322b169c789b389d594b994ea7ffcbc9e24e563a3ec9b69bc3c1 quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9 2024-03-22T05:51:27.044 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:51:26 smithi186 podman[91806]: 2024-03-22 05:51:26.771707666 +0000 UTC m=+0.129569174 container create ce84ee032f57461a94106c2b2dada32330c8a199fc83fee40031941384353d73 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240319, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-22T05:51:27.044 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:51:27 smithi186 podman[91806]: 2024-03-22 05:51:27.000258924 +0000 UTC m=+0.358120444 container init ce84ee032f57461a94106c2b2dada32330c8a199fc83fee40031941384353d73 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7-deactivate, ceph=True, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-22T05:51:27.044 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:51:27 smithi186 podman[91806]: 2024-03-22 05:51:27.003515882 +0000 UTC m=+0.361377401 container start ce84ee032f57461a94106c2b2dada32330c8a199fc83fee40031941384353d73 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, io.buildah.version=1.33.5) 2024-03-22T05:51:27.342 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:26 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:51:26] "GET /metrics HTTP/1.1" 200 38274 "" "Prometheus/2.43.0" 2024-03-22T05:51:27.342 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:51:27 smithi186 podman[91806]: 2024-03-22 05:51:27.057599144 +0000 UTC m=+0.415460654 container attach ce84ee032f57461a94106c2b2dada32330c8a199fc83fee40031941384353d73 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240319) 2024-03-22T05:51:27.342 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:51:27 smithi186 podman[91806]: 2024-03-22 05:51:27.287324619 +0000 UTC m=+0.645186123 container died ce84ee032f57461a94106c2b2dada32330c8a199fc83fee40031941384353d73 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-22T05:51:27.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:27 smithi005 ceph-mon[84705]: pgmap v261: 161 pgs: 23 stale+active+clean, 138 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 0 B/s wr, 125 op/s 2024-03-22T05:51:27.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:27 smithi005 ceph-mon[89904]: pgmap v261: 161 pgs: 23 stale+active+clean, 138 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 0 B/s wr, 125 op/s 2024-03-22T05:51:27.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:27 smithi186 ceph-mon[67903]: pgmap v261: 161 pgs: 23 stale+active+clean, 138 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 0 B/s wr, 125 op/s 2024-03-22T05:51:28.294 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:51:27 smithi186 podman[91832]: 2024-03-22 05:51:27.860943367 +0000 UTC m=+0.568691967 container remove ce84ee032f57461a94106c2b2dada32330c8a199fc83fee40031941384353d73 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217) 2024-03-22T05:51:28.294 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:51:27 smithi186 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@osd.7.service: Deactivated successfully. 2024-03-22T05:51:28.294 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:51:27 smithi186 systemd[1]: Stopped Ceph osd.7 for f59f36e2-e80a-11ee-95cd-87774f69a715. 2024-03-22T05:51:28.294 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:51:27 smithi186 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@osd.7.service: Consumed 17.603s CPU time. 2024-03-22T05:51:28.294 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:51:28 smithi186 systemd[1]: Starting Ceph osd.7 for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:51:28.664 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:51:28 smithi186 podman[91940]: 2024-03-22 05:51:28.206753733 +0000 UTC m=+0.019197809 image pull aa461e7c5e7f322b169c789b389d594b994ea7ffcbc9e24e563a3ec9b69bc3c1 quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9 2024-03-22T05:51:28.664 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:51:28 smithi186 podman[91940]: 2024-03-22 05:51:28.404109641 +0000 UTC m=+0.216553705 container create 6a2ed2d18692212a4237d550c93a10a35bf5fb2a4b225376a4fc72a936e3f5c7 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7-activate, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-22T05:51:28.664 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:51:28 smithi186 podman[91940]: 2024-03-22 05:51:28.664996838 +0000 UTC m=+0.477440916 container init 6a2ed2d18692212a4237d550c93a10a35bf5fb2a4b225376a4fc72a936e3f5c7 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7-activate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217) 2024-03-22T05:51:28.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:28 smithi005 ceph-mon[84705]: pgmap v263: 161 pgs: 11 active+undersized+degraded, 10 peering, 11 stale+active+clean, 26 active+undersized, 103 active+clean; 585 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 89 op/s; 34/723 objects degraded (4.703%) 2024-03-22T05:51:28.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:28 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:28.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:28 smithi005 ceph-mon[84705]: Health check failed: Degraded data redundancy: 34/723 objects degraded (4.703%), 11 pgs degraded (PG_DEGRADED) 2024-03-22T05:51:28.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:28 smithi005 ceph-mon[89904]: pgmap v263: 161 pgs: 11 active+undersized+degraded, 10 peering, 11 stale+active+clean, 26 active+undersized, 103 active+clean; 585 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 89 op/s; 34/723 objects degraded (4.703%) 2024-03-22T05:51:28.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:28 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:28.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:28 smithi005 ceph-mon[89904]: Health check failed: Degraded data redundancy: 34/723 objects degraded (4.703%), 11 pgs degraded (PG_DEGRADED) 2024-03-22T05:51:29.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:28 smithi186 ceph-mon[67903]: pgmap v263: 161 pgs: 11 active+undersized+degraded, 10 peering, 11 stale+active+clean, 26 active+undersized, 103 active+clean; 585 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 0 B/s wr, 89 op/s; 34/723 objects degraded (4.703%) 2024-03-22T05:51:29.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:28 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:29.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:28 smithi186 ceph-mon[67903]: Health check failed: Degraded data redundancy: 34/723 objects degraded (4.703%), 11 pgs degraded (PG_DEGRADED) 2024-03-22T05:51:29.044 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:51:28 smithi186 podman[91940]: 2024-03-22 05:51:28.668456155 +0000 UTC m=+0.480900223 container start 6a2ed2d18692212a4237d550c93a10a35bf5fb2a4b225376a4fc72a936e3f5c7 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7-activate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS) 2024-03-22T05:51:29.044 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:51:28 smithi186 podman[91940]: 2024-03-22 05:51:28.740138294 +0000 UTC m=+0.552582364 container attach 6a2ed2d18692212a4237d550c93a10a35bf5fb2a4b225376a4fc72a936e3f5c7 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7-activate, CEPH_POINT_RELEASE=, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-22T05:51:29.410 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:51:29 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7-activate[91955]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-22T05:51:29.410 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:51:29 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7-activate[91955]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-03-22T05:51:29.410 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:51:29 smithi186 bash[91940]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-22T05:51:29.411 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:51:29 smithi186 bash[91940]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-03-22T05:51:29.411 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:51:29 smithi186 bash[91940]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-03-22T05:51:29.411 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:51:29 smithi186 bash[91940]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-03-22T05:51:29.411 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:51:29 smithi186 bash[91940]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-03-22T05:51:29.411 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:51:29 smithi186 bash[91940]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-22T05:51:29.411 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:51:29 smithi186 bash[91940]: --> ceph-volume raw activate successful for osd ID: 7 2024-03-22T05:51:29.411 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:51:29 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7-activate[91955]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-03-22T05:51:29.411 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:51:29 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7-activate[91955]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-03-22T05:51:29.411 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:51:29 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7-activate[91955]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-03-22T05:51:29.411 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:51:29 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7-activate[91955]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-22T05:51:29.411 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:51:29 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7-activate[91955]: --> ceph-volume raw activate successful for osd ID: 7 2024-03-22T05:51:29.411 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:51:29 smithi186 podman[91940]: 2024-03-22 05:51:29.143511326 +0000 UTC m=+0.955955390 container died 6a2ed2d18692212a4237d550c93a10a35bf5fb2a4b225376a4fc72a936e3f5c7 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7-activate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240319, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-22T05:51:29.794 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:51:29 smithi186 podman[92077]: 2024-03-22 05:51:29.660124332 +0000 UTC m=+0.510913185 container remove 6a2ed2d18692212a4237d550c93a10a35bf5fb2a4b225376a4fc72a936e3f5c7 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7-activate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217) 2024-03-22T05:51:30.154 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:51:29 smithi186 podman[92140]: 2024-03-22 05:51:29.779166056 +0000 UTC m=+0.019057948 image pull aa461e7c5e7f322b169c789b389d594b994ea7ffcbc9e24e563a3ec9b69bc3c1 quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9 2024-03-22T05:51:30.154 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:51:29 smithi186 podman[92140]: 2024-03-22 05:51:29.919545354 +0000 UTC m=+0.159437237 container create fcb46ba843ae84db4a12ed92a080112ef8c5ff9eee34fbe202f32c91a27f2c3d (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217) 2024-03-22T05:51:30.438 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:51:30 smithi186 podman[92140]: 2024-03-22 05:51:30.154917526 +0000 UTC m=+0.394809407 container init fcb46ba843ae84db4a12ed92a080112ef8c5ff9eee34fbe202f32c91a27f2c3d (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, org.label-schema.build-date=20240319, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-22T05:51:30.438 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:51:30 smithi186 podman[92140]: 2024-03-22 05:51:30.158045056 +0000 UTC m=+0.397936948 container start fcb46ba843ae84db4a12ed92a080112ef8c5ff9eee34fbe202f32c91a27f2c3d (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319) 2024-03-22T05:51:30.438 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:51:30 smithi186 ceph-osd[92159]: -- 172.21.15.186:0/47356790 <== mon.2 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x560a4ec29680 con 0x560a4ebf2c00 2024-03-22T05:51:30.438 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:51:30 smithi186 bash[92140]: fcb46ba843ae84db4a12ed92a080112ef8c5ff9eee34fbe202f32c91a27f2c3d 2024-03-22T05:51:30.704 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:30 smithi186 ceph-mon[67903]: pgmap v264: 161 pgs: 23 active+undersized+degraded, 10 peering, 36 active+undersized, 92 active+clean; 585 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 56 op/s; 80/723 objects degraded (11.065%) 2024-03-22T05:51:30.704 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:30 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:30.704 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:30 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:30.704 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:30 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:51:30.704 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:51:30 smithi186 systemd[1]: Started Ceph osd.7 for f59f36e2-e80a-11ee-95cd-87774f69a715. 2024-03-22T05:51:30.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:30 smithi005 ceph-mon[84705]: pgmap v264: 161 pgs: 23 active+undersized+degraded, 10 peering, 36 active+undersized, 92 active+clean; 585 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 56 op/s; 80/723 objects degraded (11.065%) 2024-03-22T05:51:30.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:30 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:30.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:30 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:30.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:30 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:51:30.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:30 smithi005 ceph-mon[89904]: pgmap v264: 161 pgs: 23 active+undersized+degraded, 10 peering, 36 active+undersized, 92 active+clean; 585 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 56 op/s; 80/723 objects degraded (11.065%) 2024-03-22T05:51:30.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:30 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:30.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:30 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:30.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:30 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:51:31.043 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:51:30 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:51:30.703+0000 7f53bcaf87c0 -1 Falling back to public interface 2024-03-22T05:51:32.920 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:32 smithi186 ceph-mon[67903]: pgmap v265: 161 pgs: 28 active+undersized+degraded, 41 active+undersized, 92 active+clean; 585 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s; 97/723 objects degraded (13.416%) 2024-03-22T05:51:32.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:32 smithi005 ceph-mon[84705]: pgmap v265: 161 pgs: 28 active+undersized+degraded, 41 active+undersized, 92 active+clean; 585 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s; 97/723 objects degraded (13.416%) 2024-03-22T05:51:32.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:32 smithi005 ceph-mon[89904]: pgmap v265: 161 pgs: 28 active+undersized+degraded, 41 active+undersized, 92 active+clean; 585 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s; 97/723 objects degraded (13.416%) 2024-03-22T05:51:33.676 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:51:33 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:51:33.460+0000 7f53bcaf87c0 -1 osd.7 0 read_superblock omap replica is missing. 2024-03-22T05:51:33.677 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:51:33 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:51:33.492+0000 7f53bcaf87c0 -1 osd.7 140 log_to_monitors true 2024-03-22T05:51:33.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:33 smithi005 ceph-mon[89904]: from='osd.7 [v2:172.21.15.186:6824/69665651,v1:172.21.15.186:6825/69665651]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-22T05:51:33.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:33 smithi005 ceph-mon[89904]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-22T05:51:33.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:33 smithi005 ceph-mon[84705]: from='osd.7 [v2:172.21.15.186:6824/69665651,v1:172.21.15.186:6825/69665651]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-22T05:51:33.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:33 smithi005 ceph-mon[84705]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-22T05:51:34.044 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:51:33 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:51:33.697+0000 7f53adf50700 -1 osd.7 140 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-22T05:51:34.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:33 smithi186 ceph-mon[67903]: from='osd.7 [v2:172.21.15.186:6824/69665651,v1:172.21.15.186:6825/69665651]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-22T05:51:34.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:33 smithi186 ceph-mon[67903]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-22T05:51:34.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:34 smithi005 ceph-mon[84705]: pgmap v266: 161 pgs: 28 active+undersized+degraded, 41 active+undersized, 92 active+clean; 585 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 97/723 objects degraded (13.416%) 2024-03-22T05:51:34.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:34 smithi005 ceph-mon[84705]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-22T05:51:34.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:34 smithi005 ceph-mon[84705]: osdmap e143: 8 total, 7 up, 8 in 2024-03-22T05:51:34.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:34 smithi005 ceph-mon[84705]: from='osd.7 [v2:172.21.15.186:6824/69665651,v1:172.21.15.186:6825/69665651]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-03-22T05:51:34.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:34 smithi005 ceph-mon[84705]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-03-22T05:51:34.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:34 smithi005 ceph-mon[84705]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:51:34.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:34 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:34.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:34 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:34.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:34 smithi005 ceph-mon[89904]: pgmap v266: 161 pgs: 28 active+undersized+degraded, 41 active+undersized, 92 active+clean; 585 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 97/723 objects degraded (13.416%) 2024-03-22T05:51:34.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:34 smithi005 ceph-mon[89904]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-22T05:51:34.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:34 smithi005 ceph-mon[89904]: osdmap e143: 8 total, 7 up, 8 in 2024-03-22T05:51:34.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:34 smithi005 ceph-mon[89904]: from='osd.7 [v2:172.21.15.186:6824/69665651,v1:172.21.15.186:6825/69665651]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-03-22T05:51:34.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:34 smithi005 ceph-mon[89904]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-03-22T05:51:34.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:34 smithi005 ceph-mon[89904]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:51:34.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:34 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:34.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:34 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:35.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:34 smithi186 ceph-mon[67903]: pgmap v266: 161 pgs: 28 active+undersized+degraded, 41 active+undersized, 92 active+clean; 585 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 97/723 objects degraded (13.416%) 2024-03-22T05:51:35.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:34 smithi186 ceph-mon[67903]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-22T05:51:35.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:34 smithi186 ceph-mon[67903]: osdmap e143: 8 total, 7 up, 8 in 2024-03-22T05:51:35.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:34 smithi186 ceph-mon[67903]: from='osd.7 [v2:172.21.15.186:6824/69665651,v1:172.21.15.186:6825/69665651]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-03-22T05:51:35.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:34 smithi186 ceph-mon[67903]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-03-22T05:51:35.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:34 smithi186 ceph-mon[67903]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:51:35.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:34 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:35.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:34 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:35.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:35 smithi005 ceph-mon[84705]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-22T05:51:35.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:35 smithi005 ceph-mon[84705]: osd.7 [v2:172.21.15.186:6824/69665651,v1:172.21.15.186:6825/69665651] boot 2024-03-22T05:51:35.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:35 smithi005 ceph-mon[84705]: osdmap e144: 8 total, 8 up, 8 in 2024-03-22T05:51:35.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:35 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-22T05:51:35.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:35 smithi005 ceph-mon[84705]: osdmap e145: 8 total, 8 up, 8 in 2024-03-22T05:51:35.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:35 smithi005 ceph-mon[89904]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-22T05:51:35.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:35 smithi005 ceph-mon[89904]: osd.7 [v2:172.21.15.186:6824/69665651,v1:172.21.15.186:6825/69665651] boot 2024-03-22T05:51:35.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:35 smithi005 ceph-mon[89904]: osdmap e144: 8 total, 8 up, 8 in 2024-03-22T05:51:35.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:35 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-22T05:51:35.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:35 smithi005 ceph-mon[89904]: osdmap e145: 8 total, 8 up, 8 in 2024-03-22T05:51:36.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:35 smithi186 ceph-mon[67903]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-22T05:51:36.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:35 smithi186 ceph-mon[67903]: osd.7 [v2:172.21.15.186:6824/69665651,v1:172.21.15.186:6825/69665651] boot 2024-03-22T05:51:36.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:35 smithi186 ceph-mon[67903]: osdmap e144: 8 total, 8 up, 8 in 2024-03-22T05:51:36.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:35 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-22T05:51:36.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:35 smithi186 ceph-mon[67903]: osdmap e145: 8 total, 8 up, 8 in 2024-03-22T05:51:36.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:36 smithi005 ceph-mon[89904]: pgmap v269: 161 pgs: 28 active+undersized+degraded, 41 active+undersized, 92 active+clean; 585 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 97/723 objects degraded (13.416%) 2024-03-22T05:51:36.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:36 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:36.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:36 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:36.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:36 smithi005 ceph-mon[84705]: pgmap v269: 161 pgs: 28 active+undersized+degraded, 41 active+undersized, 92 active+clean; 585 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 97/723 objects degraded (13.416%) 2024-03-22T05:51:36.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:36 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:36.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:36 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:37.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:36 smithi186 ceph-mon[67903]: pgmap v269: 161 pgs: 28 active+undersized+degraded, 41 active+undersized, 92 active+clean; 585 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 97/723 objects degraded (13.416%) 2024-03-22T05:51:37.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:36 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:37.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:36 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:37.452 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:36 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:51:36] "GET /metrics HTTP/1.1" 200 38274 "" "Prometheus/2.43.0" 2024-03-22T05:51:37.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:37 smithi005 ceph-mon[84705]: Health check update: Degraded data redundancy: 97/723 objects degraded (13.416%), 28 pgs degraded (PG_DEGRADED) 2024-03-22T05:51:37.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:37 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:37.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:37 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:51:37.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:37 smithi005 ceph-mon[89904]: Health check update: Degraded data redundancy: 97/723 objects degraded (13.416%), 28 pgs degraded (PG_DEGRADED) 2024-03-22T05:51:37.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:37 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:37.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:37 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:51:38.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:37 smithi186 ceph-mon[67903]: Health check update: Degraded data redundancy: 97/723 objects degraded (13.416%), 28 pgs degraded (PG_DEGRADED) 2024-03-22T05:51:38.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:37 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:38.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:37 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:51:38.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:38 smithi005 ceph-mon[84705]: pgmap v271: 161 pgs: 15 active+undersized+degraded, 19 active+undersized, 127 active+clean; 585 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 61/723 objects degraded (8.437%) 2024-03-22T05:51:38.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:38 smithi005 ceph-mon[89904]: pgmap v271: 161 pgs: 15 active+undersized+degraded, 19 active+undersized, 127 active+clean; 585 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 61/723 objects degraded (8.437%) 2024-03-22T05:51:39.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:38 smithi186 ceph-mon[67903]: pgmap v271: 161 pgs: 15 active+undersized+degraded, 19 active+undersized, 127 active+clean; 585 KiB data, 256 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 61/723 objects degraded (8.437%) 2024-03-22T05:51:40.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:40 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:40.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:40 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:40.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:40 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:51:40.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:40 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:51:40.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:40 smithi005 ceph-mon[84705]: pgmap v272: 161 pgs: 3 active+undersized+degraded, 1 active+undersized, 157 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 916 B/s rd, 0 op/s; 4/723 objects degraded (0.553%) 2024-03-22T05:51:40.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:40 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:40.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:40 smithi005 ceph-mon[84705]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:51:40.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:40 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:51:40.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:40 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:51:40.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:40 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:51:40.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:40 smithi005 ceph-mon[84705]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:51:40.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:40 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:40.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:40 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:40.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:40 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:51:40.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:40 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:51:40.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:40 smithi005 ceph-mon[89904]: pgmap v272: 161 pgs: 3 active+undersized+degraded, 1 active+undersized, 157 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 916 B/s rd, 0 op/s; 4/723 objects degraded (0.553%) 2024-03-22T05:51:40.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:40 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:40.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:40 smithi005 ceph-mon[89904]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:51:40.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:40 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:51:40.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:40 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:51:40.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:40 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:51:40.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:40 smithi005 ceph-mon[89904]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:51:40.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:40 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:40.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:40 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:40.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:40 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:51:40.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:40 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:51:40.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:40 smithi186 ceph-mon[67903]: pgmap v272: 161 pgs: 3 active+undersized+degraded, 1 active+undersized, 157 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 916 B/s rd, 0 op/s; 4/723 objects degraded (0.553%) 2024-03-22T05:51:40.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:40 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:40.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:40 smithi186 ceph-mon[67903]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:51:40.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:40 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:51:40.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:40 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:51:40.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:40 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:51:40.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:40 smithi186 ceph-mon[67903]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:51:41.276 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:51:41.216+0000 7f7f0ecc9700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:51:41.276 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:51:41.276 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:51:41.276 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:51:41.276 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:51:41.276 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:51:41.276 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:51:41.276 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:51:41.276 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:51:41.276 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:51:41.276 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:51:41.276 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:51:41.277 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:51:41.277 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:51:41.277 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:51:41.277 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:51:41.277 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:51:41.277 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:51:41.277 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:51:41.277 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:51:41.278 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:51:41.278 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:51:41.278 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:51:41.278 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:51:41.278 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:51:41.278 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:51:41.278 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:51:41.278 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:51:41.278 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:51:41.278 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return future.result(timeout) 2024-03-22T05:51:41.278 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:51:41.278 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.__get_result() 2024-03-22T05:51:41.278 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:51:41.279 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: raise self._exception 2024-03-22T05:51:41.279 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:51:41.279 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: config_blobs=daemon_spec.final_config, 2024-03-22T05:51:41.279 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:51:41.279 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:51:41.279 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:51:41.279 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:51:41.279 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:51:41.279 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:51:41.279 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:51:41.279 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:51:41.279 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:51:41.279 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:51:41.279 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:51:41.279 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:51:41.279 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:51:41.280 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:51:41.280 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:51:41.280 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:51:41.280 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:51:41.280 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:51:41.280 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:51:41.280 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:51:41.280 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:51:41.280 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:51:41.280 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:51:41.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:41 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:51:41.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:41 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:51:41.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:41 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:51:41.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:41 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:51:41.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:41 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:51:41.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:41 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:41.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:41 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-03-22T05:51:41.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:41 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-03-22T05:51:41.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:41 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-03-22T05:51:41.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:41 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-03-22T05:51:41.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:41 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-03-22T05:51:41.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:41 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-03-22T05:51:41.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:41 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-03-22T05:51:41.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:41 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-03-22T05:51:41.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:41 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-03-22T05:51:41.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:41 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-03-22T05:51:41.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:41 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-03-22T05:51:41.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:41 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-03-22T05:51:41.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:41 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-03-22T05:51:41.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:41 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-03-22T05:51:41.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:41 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-03-22T05:51:41.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:41 smithi186 ceph-mon[67903]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4/723 objects degraded (0.553%), 3 pgs degraded) 2024-03-22T05:51:41.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:41 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-03-22T05:51:41.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:41 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-03-22T05:51:41.546 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:41 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-03-22T05:51:41.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:51:41.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:51:41.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:51:41.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:51:41.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:51:41.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:41.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-03-22T05:51:41.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-03-22T05:51:41.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-03-22T05:51:41.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-03-22T05:51:41.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-03-22T05:51:41.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-03-22T05:51:41.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-03-22T05:51:41.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-03-22T05:51:41.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-03-22T05:51:41.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-03-22T05:51:41.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-03-22T05:51:41.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-03-22T05:51:41.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-03-22T05:51:41.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-03-22T05:51:41.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-03-22T05:51:41.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-mon[84705]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4/723 objects degraded (0.553%), 3 pgs degraded) 2024-03-22T05:51:41.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-03-22T05:51:41.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-03-22T05:51:41.704 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-03-22T05:51:41.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:51:41.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:51:41.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:51:41.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:51:41.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:51:41.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:41.705 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-03-22T05:51:41.705 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-03-22T05:51:41.705 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-03-22T05:51:41.705 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-03-22T05:51:41.705 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-03-22T05:51:41.705 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-03-22T05:51:41.705 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-03-22T05:51:41.705 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-03-22T05:51:41.705 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-03-22T05:51:41.705 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-03-22T05:51:41.705 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-03-22T05:51:41.705 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-03-22T05:51:41.705 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-03-22T05:51:41.706 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-03-22T05:51:41.706 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-03-22T05:51:41.706 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-mon[89904]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4/723 objects degraded (0.553%), 3 pgs degraded) 2024-03-22T05:51:41.706 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-03-22T05:51:41.706 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-03-22T05:51:41.706 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:41 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-03-22T05:51:42.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:42 smithi186 ceph-mon[67903]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 679 B/s rd, 0 op/s 2024-03-22T05:51:42.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:42 smithi186 ceph-mon[67903]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:51:42.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:51:42.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:51:42.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:51:42.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:51:42.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:51:42.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:51:42.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:51:42.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:51:42.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:51:42.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:51:42.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:51:42.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:51:42.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:51:42.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:51:42.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:51:42.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:51:42.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:51:42.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:51:42.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:51:42.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:51:42.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:51:42.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:51:42.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:51:42.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:51:42.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:51:42.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:51:42.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:51:42.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:51:42.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: return future.result(timeout) 2024-03-22T05:51:42.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:51:42.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.__get_result() 2024-03-22T05:51:42.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:51:42.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: raise self._exception 2024-03-22T05:51:42.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:51:42.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:51:42.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:51:42.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:51:42.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:51:42.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:51:42.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:51:42.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:51:42.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:51:42.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:51:42.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:51:42.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:51:42.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:51:42.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:51:42.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:51:42.549 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:51:42.549 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:51:42.549 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:51:42.549 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:51:42.549 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:51:42.549 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:51:42.549 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:51:42.549 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:51:42.549 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:51:42.549 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:51:42.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:42 smithi186 ceph-mon[67903]: Upgrade: Setting container_image for all osd 2024-03-22T05:51:42.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:42 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-03-22T05:51:42.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:42 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-03-22T05:51:42.550 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:42 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-03-22T05:51:42.550 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:42 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-03-22T05:51:42.550 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:42 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-03-22T05:51:42.550 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:42 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-03-22T05:51:42.550 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:42 smithi186 ceph-mon[67903]: Upgrade: Setting require_osd_release to 19 squid 2024-03-22T05:51:42.550 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:42 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-03-22T05:51:42.550 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:42 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-03-22T05:51:42.604 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:42 smithi005 ceph-mon[84705]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 679 B/s rd, 0 op/s 2024-03-22T05:51:42.604 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:42 smithi005 ceph-mon[84705]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:51:42.604 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:51:42.604 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:51:42.604 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:51:42.604 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:51:42.604 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:51:42.604 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:51:42.604 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:51:42.605 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:51:42.605 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:51:42.605 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:51:42.605 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:51:42.605 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:51:42.605 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:51:42.605 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:51:42.605 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:51:42.605 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:51:42.605 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:51:42.605 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:51:42.605 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:51:42.605 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:51:42.606 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:51:42.606 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:51:42.606 INFO:journalctl@ceph.mon.a.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:51:42.606 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:51:42.606 INFO:journalctl@ceph.mon.a.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:51:42.606 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:51:42.606 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:51:42.606 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:51:42.606 INFO:journalctl@ceph.mon.a.smithi005.stdout: return future.result(timeout) 2024-03-22T05:51:42.606 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:51:42.606 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.__get_result() 2024-03-22T05:51:42.606 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:51:42.607 INFO:journalctl@ceph.mon.a.smithi005.stdout: raise self._exception 2024-03-22T05:51:42.607 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:51:42.607 INFO:journalctl@ceph.mon.a.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:51:42.607 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:51:42.607 INFO:journalctl@ceph.mon.a.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:51:42.607 INFO:journalctl@ceph.mon.a.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:51:42.607 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:51:42.607 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:51:42.607 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:51:42.607 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:51:42.607 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:51:42.607 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:51:42.608 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:51:42.608 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:51:42.608 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:51:42.608 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:51:42.608 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:51:42.608 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:51:42.608 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:51:42.608 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:51:42.608 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:51:42.609 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:42 smithi005 ceph-mon[89904]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 679 B/s rd, 0 op/s 2024-03-22T05:51:42.609 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:42 smithi005 ceph-mon[89904]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:51:42.609 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:51:42.609 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:51:42.609 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:51:42.609 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:51:42.609 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:51:42.609 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:51:42.609 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:51:42.609 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:51:42.609 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:51:42.610 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:51:42.610 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:51:42.610 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:51:42.610 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:51:42.610 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:51:42.610 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:51:42.610 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:51:42.610 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:51:42.610 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:51:42.610 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:51:42.610 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:51:42.610 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:51:42.611 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:51:42.611 INFO:journalctl@ceph.mon.c.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:51:42.611 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:51:42.611 INFO:journalctl@ceph.mon.c.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:51:42.611 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:51:42.611 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:51:42.611 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:51:42.611 INFO:journalctl@ceph.mon.c.smithi005.stdout: return future.result(timeout) 2024-03-22T05:51:42.611 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:51:42.611 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.__get_result() 2024-03-22T05:51:42.611 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:51:42.611 INFO:journalctl@ceph.mon.c.smithi005.stdout: raise self._exception 2024-03-22T05:51:42.612 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:51:42.612 INFO:journalctl@ceph.mon.c.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:51:42.612 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:51:42.612 INFO:journalctl@ceph.mon.c.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:51:42.612 INFO:journalctl@ceph.mon.c.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:51:42.612 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:51:42.612 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:51:42.612 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:51:42.612 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:51:42.612 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:51:42.612 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:51:42.612 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:51:42.612 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:51:42.613 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:51:42.613 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:51:42.613 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:51:42.613 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:51:42.613 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:51:42.613 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:51:42.613 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:51:42.613 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:51:42.613 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:51:42.614 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:51:42.614 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:51:42.614 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:51:42.614 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:42 smithi005 ceph-mon[84705]: Upgrade: Setting container_image for all osd 2024-03-22T05:51:42.614 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:42 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-03-22T05:51:42.614 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:42 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-03-22T05:51:42.614 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:42 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-03-22T05:51:42.614 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:42 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-03-22T05:51:42.614 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:42 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-03-22T05:51:42.614 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:42 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-03-22T05:51:42.614 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:42 smithi005 ceph-mon[84705]: Upgrade: Setting require_osd_release to 19 squid 2024-03-22T05:51:42.614 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:42 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-03-22T05:51:42.615 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:42 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-03-22T05:51:42.615 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:51:42.615 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:51:42.615 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:51:42.615 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:51:42.615 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:51:42.615 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:42 smithi005 ceph-mon[89904]: Upgrade: Setting container_image for all osd 2024-03-22T05:51:42.615 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:42 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-03-22T05:51:42.615 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:42 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-03-22T05:51:42.616 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:42 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-03-22T05:51:42.616 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:42 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-03-22T05:51:42.616 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:42 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-03-22T05:51:42.616 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:42 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-03-22T05:51:42.616 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:42 smithi005 ceph-mon[89904]: Upgrade: Setting require_osd_release to 19 squid 2024-03-22T05:51:42.616 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:42 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-03-22T05:51:42.616 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:42 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-03-22T05:51:43.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:43 smithi186 ceph-mon[67903]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-03-22T05:51:43.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:43 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-03-22T05:51:43.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:43 smithi186 ceph-mon[67903]: osdmap e146: 8 total, 8 up, 8 in 2024-03-22T05:51:43.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:43 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:51:43.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:43 smithi186 ceph-mon[67903]: Upgrade: Setting container_image for all mds 2024-03-22T05:51:43.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:43 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:43.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:43 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:51:43.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:43 smithi186 ceph-mon[67903]: Upgrade: Setting container_image for all rbd-mirror 2024-03-22T05:51:43.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:43 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:43.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:43 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:51:43.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:43 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:51:43.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:43 smithi186 ceph-mon[67903]: Upgrade: Setting container_image for all ceph-exporter 2024-03-22T05:51:43.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:43 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:43.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:43 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:51:43.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:43 smithi186 ceph-mon[67903]: Upgrade: Setting container_image for all nfs 2024-03-22T05:51:43.546 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:43 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:43.546 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:43 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:51:43.546 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:43 smithi186 ceph-mon[67903]: Upgrade: Setting container_image for all nvmeof 2024-03-22T05:51:43.546 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:43 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:43.546 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:43 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:51:43.546 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:43 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:51:43.546 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:43 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:51:43.546 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:43 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:51:43.546 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:43 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:51:43.546 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:43 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:51:43.546 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:43 smithi186 ceph-mon[67903]: Upgrade: Finalizing container_image settings 2024-03-22T05:51:43.546 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:43 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-22T05:51:43.546 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:43 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-22T05:51:43.547 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:43 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-22T05:51:43.547 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:43 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:51:43.547 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:43 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:51:43.547 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:43 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-22T05:51:43.547 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:43 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-22T05:51:43.547 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:43 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-22T05:51:43.547 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:43 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-22T05:51:43.547 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:43 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-22T05:51:43.547 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:43 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-22T05:51:43.547 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:43 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-03-22T05:51:43.547 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:43 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-22T05:51:43.547 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:43 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-22T05:51:43.547 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:43 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-22T05:51:43.547 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:43 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-22T05:51:43.547 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:43 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-22T05:51:43.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:43 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-22T05:51:43.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:43 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-22T05:51:43.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:43 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-22T05:51:43.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:43 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:51:43.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:43 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:51:43.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:43 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-22T05:51:43.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:43 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-22T05:51:43.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:43 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-22T05:51:43.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:43 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-22T05:51:43.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:43 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-22T05:51:43.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:43 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-22T05:51:43.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:43 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-22T05:51:43.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:43 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-22T05:51:43.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:43 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-22T05:51:43.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:43 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-22T05:51:43.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:43 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-22T05:51:43.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:43 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:51:43.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:43 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:51:43.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:43 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:51:43.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:43 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:51:43.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:43 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:51:43.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:43 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:51:43.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:43 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:51:43.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:43 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:51:43.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:43 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:51:43.550 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:43 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:51:43.550 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:43 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:51:43.550 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:43 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:51:43.550 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:43 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-22T05:51:43.550 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:43 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-22T05:51:43.550 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:43 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-22T05:51:43.550 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:43 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:51:43.550 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:43 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:43.550 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:43 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:51:43.550 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:43 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:51:43.550 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:43 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:43.550 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:43 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:51:43.550 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:43 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:51:43.550 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:43 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:51:43.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[89904]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-03-22T05:51:43.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-03-22T05:51:43.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[89904]: osdmap e146: 8 total, 8 up, 8 in 2024-03-22T05:51:43.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:51:43.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[89904]: Upgrade: Setting container_image for all mds 2024-03-22T05:51:43.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:43.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:51:43.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[89904]: Upgrade: Setting container_image for all rbd-mirror 2024-03-22T05:51:43.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:43.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:51:43.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:51:43.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[89904]: Upgrade: Setting container_image for all ceph-exporter 2024-03-22T05:51:43.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:43.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:51:43.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[89904]: Upgrade: Setting container_image for all nfs 2024-03-22T05:51:43.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:43.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:51:43.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[89904]: Upgrade: Setting container_image for all nvmeof 2024-03-22T05:51:43.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:43.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:51:43.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:51:43.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:51:43.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:51:43.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:51:43.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:51:43.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[89904]: Upgrade: Finalizing container_image settings 2024-03-22T05:51:43.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-22T05:51:43.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-22T05:51:43.705 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-22T05:51:43.705 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:51:43.705 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:51:43.705 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-22T05:51:43.705 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[84705]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-03-22T05:51:43.705 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-03-22T05:51:43.705 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[84705]: osdmap e146: 8 total, 8 up, 8 in 2024-03-22T05:51:43.705 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:51:43.705 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[84705]: Upgrade: Setting container_image for all mds 2024-03-22T05:51:43.705 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:43.705 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:51:43.705 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[84705]: Upgrade: Setting container_image for all rbd-mirror 2024-03-22T05:51:43.705 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:43.706 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:51:43.706 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:51:43.706 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[84705]: Upgrade: Setting container_image for all ceph-exporter 2024-03-22T05:51:43.706 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:43.706 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:51:43.706 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[84705]: Upgrade: Setting container_image for all nfs 2024-03-22T05:51:43.706 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:43.706 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:51:43.706 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[84705]: Upgrade: Setting container_image for all nvmeof 2024-03-22T05:51:43.706 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:43.706 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:51:43.706 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:51:43.706 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:51:43.706 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:51:43.706 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:51:43.706 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:51:43.707 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[84705]: Upgrade: Finalizing container_image settings 2024-03-22T05:51:43.707 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-22T05:51:43.707 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-22T05:51:43.707 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-22T05:51:43.707 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:51:43.707 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:51:43.707 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-22T05:51:43.707 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-22T05:51:43.707 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-22T05:51:43.707 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-22T05:51:43.707 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-22T05:51:43.707 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-22T05:51:43.707 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-03-22T05:51:43.708 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-22T05:51:43.708 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-22T05:51:43.708 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-22T05:51:43.708 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-22T05:51:43.708 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-22T05:51:43.708 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-22T05:51:43.708 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-22T05:51:43.708 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-22T05:51:43.708 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:51:43.708 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:51:43.708 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-22T05:51:43.708 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-22T05:51:43.708 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-22T05:51:43.708 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-22T05:51:43.708 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-22T05:51:43.709 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-22T05:51:43.709 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-22T05:51:43.709 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-22T05:51:43.709 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-22T05:51:43.709 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-22T05:51:43.709 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-22T05:51:43.709 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-22T05:51:43.709 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-22T05:51:43.710 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-03-22T05:51:43.710 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-22T05:51:43.710 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-22T05:51:43.710 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-22T05:51:43.710 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-22T05:51:43.710 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-22T05:51:43.710 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-22T05:51:43.710 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-22T05:51:43.710 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-22T05:51:43.710 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:51:43.710 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:51:43.710 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-22T05:51:43.710 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-22T05:51:43.710 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-22T05:51:43.710 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-22T05:51:43.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-22T05:51:43.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-22T05:51:43.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-22T05:51:43.711 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-22T05:51:43.711 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-22T05:51:43.711 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-22T05:51:43.711 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-22T05:51:43.711 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:51:43.711 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:51:43.711 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:51:43.711 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:51:43.712 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:51:43.712 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:51:43.712 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:51:43.712 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:51:43.712 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:51:43.712 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:51:43.712 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:51:43.712 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:51:43.712 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-22T05:51:43.712 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-22T05:51:43.712 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-22T05:51:43.712 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:51:43.712 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:43.712 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:51:43.712 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:51:43.713 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:43.713 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:51:43.713 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:51:43.713 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:51:43.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-22T05:51:43.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-22T05:51:43.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-22T05:51:43.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:51:43.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:51:43.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:51:43.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:51:43.713 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:51:43.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:51:43.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:51:43.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:51:43.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:51:43.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:51:43.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:51:43.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:51:43.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-22T05:51:43.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-22T05:51:43.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-22T05:51:43.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:51:43.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:43.714 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:51:43.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:51:43.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:43.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:51:43.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:51:43.715 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:43 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:51:44.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:44 smithi005 ceph-mon[84705]: Upgrade: Complete! 2024-03-22T05:51:44.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:44 smithi005 ceph-mon[84705]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-22T05:51:44.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:44 smithi005 ceph-mon[84705]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-22T05:51:44.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:44 smithi005 ceph-mon[84705]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:51:44.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:44 smithi005 ceph-mon[84705]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:51:44.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:44 smithi005 ceph-mon[89904]: Upgrade: Complete! 2024-03-22T05:51:44.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:44 smithi005 ceph-mon[89904]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-22T05:51:44.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:44 smithi005 ceph-mon[89904]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-22T05:51:44.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:44 smithi005 ceph-mon[89904]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:51:44.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:44 smithi005 ceph-mon[89904]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:51:44.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:44 smithi186 ceph-mon[67903]: Upgrade: Complete! 2024-03-22T05:51:44.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:44 smithi186 ceph-mon[67903]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-22T05:51:44.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:44 smithi186 ceph-mon[67903]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-22T05:51:44.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:44 smithi186 ceph-mon[67903]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:51:44.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:44 smithi186 ceph-mon[67903]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:51:45.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:45 smithi186 ceph-mon[67903]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:51:45.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:45 smithi186 ceph-mon[67903]: from='client.64549 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:51:45.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:45 smithi005 ceph-mon[89904]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:51:45.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:45 smithi005 ceph-mon[89904]: from='client.64549 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:51:45.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:45 smithi005 ceph-mon[84705]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:51:45.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:45 smithi005 ceph-mon[84705]: from='client.64549 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:51:45.703 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:45 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:51:45.405+0000 7f7f0ecc9700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:51:45.703 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:45 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:51:45.703 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:45 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:51:45.703 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:45 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:51:45.703 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:45 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:51:45.704 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:45 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:51:45.704 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:45 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:51:45.704 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:45 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:51:45.704 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:45 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:51:45.704 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:45 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:51:45.704 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:45 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:51:45.704 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:45 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:51:45.704 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:45 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:51:45.704 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:45 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:51:45.704 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:45 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:51:45.704 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:45 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:51:45.704 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:45 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:51:45.704 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:45 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:51:45.704 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:45 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:51:45.705 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:45 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:51:45.705 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:45 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:51:45.705 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:45 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:51:45.705 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:45 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:51:45.705 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:45 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:51:45.705 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:45 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:51:45.705 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:45 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:51:45.705 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:45 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:51:45.705 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:45 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:51:45.705 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:45 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:51:45.705 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:45 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return future.result(timeout) 2024-03-22T05:51:45.705 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:45 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:51:45.705 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:45 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.__get_result() 2024-03-22T05:51:45.705 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:45 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:51:45.705 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:45 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: raise self._exception 2024-03-22T05:51:45.706 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:45 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:51:45.706 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:45 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: config_blobs=daemon_spec.final_config, 2024-03-22T05:51:45.706 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:45 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:51:45.706 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:45 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:51:45.706 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:45 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:51:45.706 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:45 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:51:45.706 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:45 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:51:45.706 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:45 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:51:45.706 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:45 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:51:45.706 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:45 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:51:45.706 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:45 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:51:45.706 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:45 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:51:45.706 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:45 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:51:45.706 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:45 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:51:45.706 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:45 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:51:45.707 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:45 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:51:45.707 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:45 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:51:45.707 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:45 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:51:45.707 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:45 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:51:45.707 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:45 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:51:45.707 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:45 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:51:45.707 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:45 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:51:45.707 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:45 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:51:45.707 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:45 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:51:45.709 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:45 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:51:46.176 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph orch ps' 2024-03-22T05:51:46.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:46 smithi005 ceph-mon[84705]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 689 B/s rd, 0 op/s 2024-03-22T05:51:46.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:46 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:51:46.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:46 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:51:46.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:46 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:51:46.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:46 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:46.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:46 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:51:46.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:46 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:51:46.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:46 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:51:46.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:46 smithi005 ceph-mon[89904]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 689 B/s rd, 0 op/s 2024-03-22T05:51:46.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:46 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:51:46.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:46 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:51:46.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:46 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:51:46.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:46 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:46.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:46 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:51:46.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:46 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:51:46.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:46 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:51:46.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:46 smithi186 ceph-mon[67903]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 689 B/s rd, 0 op/s 2024-03-22T05:51:46.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:46 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:51:46.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:46 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:51:46.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:46 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:51:46.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:46 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:46.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:46 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:51:46.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:46 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:51:46.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:46 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:51:47.293 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:46 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:51:46] "GET /metrics HTTP/1.1" 200 38227 "" "Prometheus/2.43.0" 2024-03-22T05:51:47.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:47 smithi186 ceph-mon[67903]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:51:47.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:51:47.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:51:47.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:51:47.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:51:47.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:51:47.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:51:47.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:51:47.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:51:47.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:51:47.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:51:47.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:51:47.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:51:47.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:51:47.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:51:47.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:51:47.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:51:47.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:51:47.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:51:47.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:51:47.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:51:47.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:51:47.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:51:47.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:51:47.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:51:47.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:51:47.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:51:47.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:51:47.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:51:47.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: return future.result(timeout) 2024-03-22T05:51:47.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:51:47.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.__get_result() 2024-03-22T05:51:47.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:51:47.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: raise self._exception 2024-03-22T05:51:47.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:51:47.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:51:47.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:51:47.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:51:47.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:51:47.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:51:47.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:51:47.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:51:47.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:51:47.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:51:47.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:51:47.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:51:47.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:51:47.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:51:47.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:51:47.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:51:47.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:51:47.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:51:47.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:51:47.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:51:47.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:51:47.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:51:47.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:51:47.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:51:47.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:51:47.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:47 smithi186 ceph-mon[67903]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 825 B/s rd, 0 op/s 2024-03-22T05:51:47.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:47 smithi186 ceph-mon[67903]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:51:47.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:47 smithi186 ceph-mon[67903]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:51:47.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:47 smithi186 ceph-mon[67903]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:51:47.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:47 smithi005 ceph-mon[84705]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:51:47.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:51:47.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:51:47.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:51:47.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:51:47.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:51:47.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:51:47.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:51:47.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:51:47.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:51:47.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:51:47.703 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:51:47.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:51:47.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:51:47.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:51:47.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:51:47.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:51:47.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:51:47.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:51:47.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:51:47.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:51:47.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:51:47.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:51:47.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:51:47.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:51:47.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:51:47.704 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:51:47.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:51:47.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:51:47.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: return future.result(timeout) 2024-03-22T05:51:47.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:51:47.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.__get_result() 2024-03-22T05:51:47.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:51:47.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: raise self._exception 2024-03-22T05:51:47.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:51:47.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:51:47.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:51:47.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:51:47.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:51:47.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:51:47.705 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:51:47.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:51:47.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:51:47.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:51:47.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:51:47.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:51:47.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:51:47.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:51:47.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:51:47.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:51:47.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:51:47.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:51:47.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:51:47.706 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:51:47.707 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:51:47.707 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:51:47.707 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:51:47.707 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:51:47.707 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:51:47.707 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:47 smithi005 ceph-mon[84705]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 825 B/s rd, 0 op/s 2024-03-22T05:51:47.707 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:47 smithi005 ceph-mon[84705]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:51:47.707 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:47 smithi005 ceph-mon[84705]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:51:47.707 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:47 smithi005 ceph-mon[84705]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:51:47.707 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:47 smithi005 ceph-mon[89904]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:51:47.707 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:51:47.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:51:47.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:51:47.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:51:47.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:51:47.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:51:47.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:51:47.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:51:47.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:51:47.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:51:47.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:51:47.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:51:47.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:51:47.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:51:47.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:51:47.708 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:51:47.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:51:47.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:51:47.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:51:47.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:51:47.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:51:47.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:51:47.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:51:47.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:51:47.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:51:47.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:51:47.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:51:47.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:51:47.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: return future.result(timeout) 2024-03-22T05:51:47.709 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:51:47.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.__get_result() 2024-03-22T05:51:47.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:51:47.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: raise self._exception 2024-03-22T05:51:47.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:51:47.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:51:47.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:51:47.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:51:47.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:51:47.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:51:47.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:51:47.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:51:47.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:51:47.710 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:51:47.711 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:51:47.711 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:51:47.711 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:51:47.711 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:51:47.711 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:51:47.711 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:51:47.711 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:51:47.711 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:51:47.711 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:51:47.711 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:51:47.711 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:51:47.711 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:51:47.711 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:51:47.711 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:51:47.711 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:51:47.712 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:47 smithi005 ceph-mon[89904]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 825 B/s rd, 0 op/s 2024-03-22T05:51:47.712 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:47 smithi005 ceph-mon[89904]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:51:47.712 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:47 smithi005 ceph-mon[89904]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:51:47.712 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:47 smithi005 ceph-mon[89904]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:51:48.453 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:51:48.222+0000 7f7f0ecc9700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:51:48.454 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:51:48.454 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:51:48.454 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:51:48.454 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:51:48.454 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:51:48.454 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:51:48.454 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:51:48.454 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:51:48.454 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:51:48.454 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:51:48.454 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:51:48.454 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:51:48.454 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:51:48.454 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:51:48.455 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:51:48.455 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:51:48.455 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:51:48.455 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:51:48.455 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:51:48.455 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:51:48.455 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:51:48.455 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:51:48.455 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:51:48.455 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:51:48.455 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:51:48.455 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:51:48.455 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:51:48.455 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:51:48.455 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return future.result(timeout) 2024-03-22T05:51:48.456 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:51:48.456 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.__get_result() 2024-03-22T05:51:48.456 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:51:48.456 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: raise self._exception 2024-03-22T05:51:48.456 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:51:48.456 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: config_blobs=daemon_spec.final_config, 2024-03-22T05:51:48.456 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:51:48.456 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:51:48.456 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:51:48.456 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:51:48.456 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:51:48.456 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:51:48.456 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:51:48.456 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:51:48.457 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:51:48.457 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:51:48.457 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:51:48.457 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:51:48.457 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:51:48.457 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:51:48.457 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:51:48.457 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:51:48.457 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:51:48.457 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:51:48.457 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:51:48.457 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:51:48.457 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:51:48.458 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:51:48.458 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:48 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:51:48.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:48 smithi186 ceph-mon[67903]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 999 B/s rd, 0 op/s 2024-03-22T05:51:48.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:48 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:48.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:48 smithi186 ceph-mon[67903]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:51:48.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:51:48.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:51:48.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:51:48.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:51:48.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:51:48.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:51:48.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:51:48.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:51:48.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:51:48.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:51:48.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:51:48.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:51:48.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:51:48.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:51:48.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:51:48.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:51:48.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:51:48.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:51:48.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:51:48.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:51:48.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:51:48.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:51:48.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:51:48.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:51:48.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:51:48.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:51:48.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:51:48.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:51:48.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: return future.result(timeout) 2024-03-22T05:51:48.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:51:48.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.__get_result() 2024-03-22T05:51:48.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:51:48.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: raise self._exception 2024-03-22T05:51:48.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:51:48.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:51:48.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:51:48.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:51:48.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:51:48.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:51:48.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:51:48.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:51:48.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:51:48.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:51:48.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:51:48.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:51:48.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:51:48.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:51:48.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:51:48.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:51:48.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:51:48.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:51:48.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:51:48.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:51:48.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:51:48.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:51:48.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:51:48.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:51:48.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:51:48.935 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:48 smithi005 ceph-mon[84705]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 999 B/s rd, 0 op/s 2024-03-22T05:51:48.935 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:48 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:48.935 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:48 smithi005 ceph-mon[84705]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:51:48.935 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:51:48.935 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:51:48.935 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:51:48.935 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:51:48.935 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:51:48.935 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:51:48.936 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:51:48.936 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:51:48.936 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:51:48.936 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:51:48.936 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:51:48.936 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:51:48.936 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:51:48.936 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:51:48.936 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:51:48.936 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:51:48.936 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:51:48.936 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:51:48.936 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:51:48.936 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:51:48.937 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:51:48.937 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:51:48.937 INFO:journalctl@ceph.mon.a.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:51:48.937 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:51:48.937 INFO:journalctl@ceph.mon.a.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:51:48.937 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:51:48.937 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:51:48.937 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:51:48.937 INFO:journalctl@ceph.mon.a.smithi005.stdout: return future.result(timeout) 2024-03-22T05:51:48.937 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:51:48.937 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.__get_result() 2024-03-22T05:51:48.937 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:51:48.937 INFO:journalctl@ceph.mon.a.smithi005.stdout: raise self._exception 2024-03-22T05:51:48.937 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:51:48.937 INFO:journalctl@ceph.mon.a.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:51:48.938 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:51:48.938 INFO:journalctl@ceph.mon.a.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:51:48.938 INFO:journalctl@ceph.mon.a.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:51:48.938 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:51:48.938 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:51:48.938 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:51:48.938 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:51:48.938 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:51:48.938 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:51:48.938 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:51:48.938 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:51:48.938 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:51:48.938 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:51:48.938 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:51:48.939 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:51:48.939 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:51:48.939 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:51:48.939 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:48 smithi005 ceph-mon[89904]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 999 B/s rd, 0 op/s 2024-03-22T05:51:48.939 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:48 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:48.939 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:48 smithi005 ceph-mon[89904]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:51:48.939 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:51:48.939 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:51:48.939 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:51:48.939 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:51:48.940 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:51:48.940 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:51:48.940 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:51:48.940 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:51:48.940 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:51:48.940 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:51:48.940 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:51:48.940 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:51:48.940 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:51:48.940 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:51:48.940 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:51:48.940 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:51:48.940 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:51:48.940 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:51:48.941 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:51:48.941 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:51:48.941 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:51:48.941 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:51:48.941 INFO:journalctl@ceph.mon.c.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:51:48.941 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:51:48.941 INFO:journalctl@ceph.mon.c.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:51:48.941 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:51:48.941 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:51:48.941 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:51:48.941 INFO:journalctl@ceph.mon.c.smithi005.stdout: return future.result(timeout) 2024-03-22T05:51:48.941 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:51:48.941 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.__get_result() 2024-03-22T05:51:48.941 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:51:48.942 INFO:journalctl@ceph.mon.c.smithi005.stdout: raise self._exception 2024-03-22T05:51:48.942 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:51:48.942 INFO:journalctl@ceph.mon.c.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:51:48.942 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:51:48.942 INFO:journalctl@ceph.mon.c.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:51:48.942 INFO:journalctl@ceph.mon.c.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:51:48.942 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:51:48.942 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:51:48.942 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:51:48.942 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:51:48.942 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:51:48.942 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:51:48.942 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:51:48.942 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:51:48.943 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:51:48.943 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:51:48.943 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:51:48.943 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:51:48.943 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:51:48.943 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:51:48.943 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:51:48.943 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:51:48.943 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:51:48.943 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:51:48.943 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:51:48.943 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:51:48.943 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:51:48.944 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:51:48.944 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:51:48.944 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:51:48.944 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:51:48.944 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:51:49.390 INFO:teuthology.orchestra.run.smithi005.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T05:51:49.390 INFO:teuthology.orchestra.run.smithi005.stdout:alertmanager.a smithi005 *:9093,9094 running (9m) 2m ago 31m 28.0M - 0.25.0 c8568f914cd2 ee99dfc435e8 2024-03-22T05:51:49.390 INFO:teuthology.orchestra.run.smithi005.stdout:grafana.a smithi186 *:3000 running (9m) 14s ago 31m 81.3M - 9.4.12 2bacad6d85d8 16fd57dbbd71 2024-03-22T05:51:49.390 INFO:teuthology.orchestra.run.smithi005.stdout:iscsi.foo.smithi005.mcdwbp smithi005 unknown 2m ago 30m 64.6M - 3.5 e1d6a67b021e 5215e03bf81b 2024-03-22T05:51:49.391 INFO:teuthology.orchestra.run.smithi005.stdout:mgr.x smithi186 *:8443,9283,8765 running (10m) 14s ago 35m 454M - 19.0.0-2299-gea01217f aa461e7c5e7f c556421a73f3 2024-03-22T05:51:49.391 INFO:teuthology.orchestra.run.smithi005.stdout:mgr.y smithi005 *:8443,9283,8765 running (20m) 2m ago 37m 511M - 19.0.0-2299-gea01217f aa461e7c5e7f 62a813eebc0d 2024-03-22T05:51:49.391 INFO:teuthology.orchestra.run.smithi005.stdout:mon.a smithi005 running (7m) 2m ago 37m 55.3M 2048M 19.0.0-2299-gea01217f aa461e7c5e7f 111a2aa24dd9 2024-03-22T05:51:49.391 INFO:teuthology.orchestra.run.smithi005.stdout:mon.b smithi186 running (7m) 14s ago 35m 53.1M 2048M 19.0.0-2299-gea01217f aa461e7c5e7f c1798feac15b 2024-03-22T05:51:49.391 INFO:teuthology.orchestra.run.smithi005.stdout:mon.c smithi005 running (6m) 2m ago 36m 42.0M 2048M 19.0.0-2299-gea01217f aa461e7c5e7f 708c72ff74f5 2024-03-22T05:51:49.391 INFO:teuthology.orchestra.run.smithi005.stdout:node-exporter.a smithi005 *:9100 running (10m) 2m ago 31m 16.8M - 1.5.0 0da6a335fe13 f635cd72b9d3 2024-03-22T05:51:49.391 INFO:teuthology.orchestra.run.smithi005.stdout:node-exporter.b smithi186 *:9100 running (9m) 14s ago 31m 14.7M - 1.5.0 0da6a335fe13 d7e0b3d20191 2024-03-22T05:51:49.391 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0 smithi005 running (4m) 2m ago 35m 44.8M 2993M 19.0.0-2299-gea01217f aa461e7c5e7f 31b6cdee5f5b 2024-03-22T05:51:49.391 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1 smithi005 running (4m) 2m ago 35m 44.2M 2993M 19.0.0-2299-gea01217f aa461e7c5e7f 7d21bc9fd3c2 2024-03-22T05:51:49.391 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2 smithi005 running (3m) 2m ago 34m 42.0M 2993M 19.0.0-2299-gea01217f aa461e7c5e7f 628204cb7c98 2024-03-22T05:51:49.391 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3 smithi005 running (2m) 2m ago 34m 14.9M 2993M 19.0.0-2299-gea01217f aa461e7c5e7f d31a01a0230c 2024-03-22T05:51:49.391 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4 smithi186 running (114s) 14s ago 33m 57.4M 3248M 19.0.0-2299-gea01217f aa461e7c5e7f 8f7619b726f3 2024-03-22T05:51:49.391 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5 smithi186 running (77s) 14s ago 33m 52.2M 3248M 19.0.0-2299-gea01217f aa461e7c5e7f b80f7eceb2d7 2024-03-22T05:51:49.392 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6 smithi186 running (41s) 14s ago 33m 51.7M 3248M 19.0.0-2299-gea01217f aa461e7c5e7f db7c81962c65 2024-03-22T05:51:49.392 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7 smithi186 running (19s) 14s ago 32m 15.1M 3248M 19.0.0-2299-gea01217f aa461e7c5e7f fcb46ba843ae 2024-03-22T05:51:49.392 INFO:teuthology.orchestra.run.smithi005.stdout:prometheus.a smithi186 *:9095 running (9m) 14s ago 32m 72.0M - 2.43.0 a07b618ecd1d a5a043588d6d 2024-03-22T05:51:49.392 INFO:teuthology.orchestra.run.smithi005.stdout:rgw.foo.smithi005.exdhqu smithi005 *:8000 running (30m) 2m ago 30m 81.5M - 17.2.0 e1d6a67b021e 6bb27fe11553 2024-03-22T05:51:49.392 INFO:teuthology.orchestra.run.smithi005.stdout:rgw.foo.smithi186.mcgodk smithi186 *:8000 running (30m) 14s ago 30m 80.2M - 17.2.0 e1d6a67b021e 19b5c20a34f7 2024-03-22T05:51:50.341 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph versions | jq -e '"'"'.osd | length == 1'"'"'' 2024-03-22T05:51:50.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:50 smithi005 ceph-mon[89904]: from='client.54621 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:51:50.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:50 smithi005 ceph-mon[89904]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 803 B/s rd, 0 op/s 2024-03-22T05:51:50.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:50 smithi005 ceph-mon[84705]: from='client.54621 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:51:50.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:50 smithi005 ceph-mon[84705]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 803 B/s rd, 0 op/s 2024-03-22T05:51:51.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:50 smithi186 ceph-mon[67903]: from='client.54621 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:51:51.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:50 smithi186 ceph-mon[67903]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 803 B/s rd, 0 op/s 2024-03-22T05:51:52.580 INFO:teuthology.orchestra.run.smithi005.stdout:true 2024-03-22T05:51:52.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:52 smithi005 ceph-mon[89904]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 611 B/s rd, 0 op/s 2024-03-22T05:51:52.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:52 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:52.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:52 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:51:52.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:52 smithi005 ceph-mon[89904]: from='client.? 172.21.15.5:0/2673943186' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:51:52.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:52 smithi005 ceph-mon[84705]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 611 B/s rd, 0 op/s 2024-03-22T05:51:52.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:52 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:52.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:52 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:51:52.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:52 smithi005 ceph-mon[84705]: from='client.? 172.21.15.5:0/2673943186' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:51:53.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:52 smithi186 ceph-mon[67903]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 611 B/s rd, 0 op/s 2024-03-22T05:51:53.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:52 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:51:53.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:52 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:51:53.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:52 smithi186 ceph-mon[67903]: from='client.? 172.21.15.5:0/2673943186' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:51:53.361 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph versions | jq -e '"'"'.osd | keys'"'"' | grep $sha1' 2024-03-22T05:51:54.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:54 smithi005 ceph-mon[84705]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:51:54.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:54 smithi005 ceph-mon[84705]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:51:54.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:54 smithi005 ceph-mon[89904]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:51:54.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:54 smithi005 ceph-mon[89904]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:51:55.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:54 smithi186 ceph-mon[67903]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:51:55.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:54 smithi186 ceph-mon[67903]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:51:55.696 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev)" 2024-03-22T05:51:55.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:55 smithi005 ceph-mon[84705]: from='client.? 172.21.15.5:0/1592682707' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:51:55.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:55 smithi005 ceph-mon[89904]: from='client.? 172.21.15.5:0/1592682707' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:51:56.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:55 smithi186 ceph-mon[67903]: from='client.? 172.21.15.5:0/1592682707' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:51:56.373 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph orch upgrade status' 2024-03-22T05:51:56.944 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:56 smithi005 ceph-mon[84705]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:51:56.944 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:56 smithi005 ceph-mon[89904]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:51:57.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:56 smithi186 ceph-mon[67903]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:51:57.202 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:51:56 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:51:56] "GET /metrics HTTP/1.1" 200 38366 "" "Prometheus/2.43.0" 2024-03-22T05:51:58.316 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-03-22T05:51:58.317 INFO:teuthology.orchestra.run.smithi005.stdout: "target_image": null, 2024-03-22T05:51:58.317 INFO:teuthology.orchestra.run.smithi005.stdout: "in_progress": false, 2024-03-22T05:51:58.317 INFO:teuthology.orchestra.run.smithi005.stdout: "which": "", 2024-03-22T05:51:58.317 INFO:teuthology.orchestra.run.smithi005.stdout: "services_complete": [], 2024-03-22T05:51:58.317 INFO:teuthology.orchestra.run.smithi005.stdout: "progress": null, 2024-03-22T05:51:58.317 INFO:teuthology.orchestra.run.smithi005.stdout: "message": "", 2024-03-22T05:51:58.317 INFO:teuthology.orchestra.run.smithi005.stdout: "is_paused": false 2024-03-22T05:51:58.317 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-03-22T05:51:58.928 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph health detail' 2024-03-22T05:51:58.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:58 smithi005 ceph-mon[84705]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:51:58.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:51:58 smithi005 ceph-mon[84705]: from='client.54639 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:51:58.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:58 smithi005 ceph-mon[89904]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:51:58.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:51:58 smithi005 ceph-mon[89904]: from='client.54639 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:51:59.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:58 smithi186 ceph-mon[67903]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:51:59.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:51:58 smithi186 ceph-mon[67903]: from='client.54639 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:52:00.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:00 smithi005 ceph-mon[89904]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:52:00.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:00 smithi005 ceph-mon[84705]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:52:01.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:00 smithi186 ceph-mon[67903]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:52:01.181 INFO:teuthology.orchestra.run.smithi005.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-22T05:52:01.181 INFO:teuthology.orchestra.run.smithi005.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-22T05:52:01.181 INFO:teuthology.orchestra.run.smithi005.stdout: daemon iscsi.foo.smithi005.mcdwbp on smithi005 is in unknown state 2024-03-22T05:52:01.904 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:01 smithi005 ceph-mon[84705]: from='client.? 172.21.15.5:0/1651157382' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T05:52:01.904 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:01 smithi005 ceph-mon[89904]: from='client.? 172.21.15.5:0/1651157382' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T05:52:01.927 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --services rgw.foo' 2024-03-22T05:52:02.002 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:01 smithi186 ceph-mon[67903]: from='client.? 172.21.15.5:0/1651157382' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T05:52:02.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:02 smithi005 ceph-mon[84705]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:52:02.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:02 smithi005 ceph-mon[89904]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:52:03.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:02 smithi186 ceph-mon[67903]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:52:05.011 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:04 smithi005 ceph-mon[84705]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:52:05.011 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:04 smithi005 ceph-mon[84705]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:52:05.011 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:04 smithi005 ceph-mon[84705]: from='client.64585 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37", "services": "rgw.foo", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:52:05.011 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:04 smithi005 ceph-mon[89904]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:52:05.012 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:04 smithi005 ceph-mon[89904]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:52:05.012 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:04 smithi005 ceph-mon[89904]: from='client.64585 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37", "services": "rgw.foo", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:52:05.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:04 smithi186 ceph-mon[67903]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:52:05.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:04 smithi186 ceph-mon[67903]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:52:05.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:04 smithi186 ceph-mon[67903]: from='client.64585 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37", "services": "rgw.foo", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:52:06.274 INFO:teuthology.orchestra.run.smithi005.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37 2024-03-22T05:52:06.945 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-03-22T05:52:07.202 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:52:06] "GET /metrics HTTP/1.1" 200 38366 "" "Prometheus/2.43.0" 2024-03-22T05:52:07.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:07 smithi186 ceph-mon[67903]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:52:07.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:07 smithi186 ceph-mon[67903]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37 2024-03-22T05:52:07.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:07 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:07.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:07 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:52:07.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:07 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:52:07.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:07 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:52:07.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:07 smithi186 ceph-mon[67903]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 942 B/s rd, 0 op/s 2024-03-22T05:52:07.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:07 smithi186 ceph-mon[67903]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 577 B/s rd, 0 op/s 2024-03-22T05:52:07.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:07 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:07.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:07 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:52:07.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:07 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:52:07.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:07 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:52:07.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:07 smithi005 ceph-mon[84705]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:52:07.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:07 smithi005 ceph-mon[84705]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37 2024-03-22T05:52:07.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:07 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:07.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:07 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:52:07.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:07 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:52:07.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:07 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:52:07.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:07 smithi005 ceph-mon[84705]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 942 B/s rd, 0 op/s 2024-03-22T05:52:07.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:07 smithi005 ceph-mon[84705]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 577 B/s rd, 0 op/s 2024-03-22T05:52:07.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:07 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:07.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:07 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:52:07.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:07 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:52:07.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:07 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:52:07.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:07 smithi005 ceph-mon[89904]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:52:07.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:07 smithi005 ceph-mon[89904]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37 2024-03-22T05:52:07.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:07 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:07.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:07 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:52:07.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:07 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:52:07.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:07 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:52:07.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:07 smithi005 ceph-mon[89904]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 942 B/s rd, 0 op/s 2024-03-22T05:52:07.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:07 smithi005 ceph-mon[89904]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 577 B/s rd, 0 op/s 2024-03-22T05:52:07.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:07 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:07.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:07 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:52:07.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:07 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:52:07.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:07 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:52:08.391 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:08 smithi005 ceph-mon[84705]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:52:08.391 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:08 smithi005 ceph-mon[84705]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:52:08.391 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:08 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:52:08.391 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:08 smithi005 ceph-mon[89904]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:52:08.391 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:08 smithi005 ceph-mon[89904]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:52:08.391 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:08 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:52:08.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:08 smithi186 ceph-mon[67903]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:52:08.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:08 smithi186 ceph-mon[67903]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:52:08.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:08 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:52:09.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:09 smithi005 ceph-mon[84705]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:52:09.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:09 smithi005 ceph-mon[89904]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:52:09.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:09 smithi186 ceph-mon[67903]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:52:10.050 INFO:teuthology.orchestra.run.smithi005.stdout:true 2024-03-22T05:52:10.050 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:09 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:52:09.910+0000 7f7f0ecc9700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:52:10.051 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:09 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:52:10.051 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:09 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:52:10.051 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:09 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:52:10.051 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:09 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:52:10.051 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:09 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:52:10.051 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:09 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:52:10.051 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:09 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:52:10.051 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:09 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:52:10.051 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:09 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:52:10.051 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:09 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:52:10.051 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:09 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:52:10.051 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:09 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:52:10.051 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:09 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:52:10.052 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:09 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:52:10.052 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:09 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:52:10.052 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:09 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:52:10.052 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:09 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:52:10.052 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:09 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:52:10.052 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:09 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:52:10.053 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:09 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:52:10.053 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:09 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:52:10.053 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:09 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:52:10.053 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:09 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:52:10.053 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:09 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:52:10.053 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:09 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:52:10.053 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:09 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:52:10.053 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:09 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:52:10.053 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:09 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:52:10.053 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:09 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return future.result(timeout) 2024-03-22T05:52:10.053 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:09 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:52:10.053 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:09 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.__get_result() 2024-03-22T05:52:10.053 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:09 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:52:10.053 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:09 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: raise self._exception 2024-03-22T05:52:10.053 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:09 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:52:10.054 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:09 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: config_blobs=daemon_spec.final_config, 2024-03-22T05:52:10.054 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:09 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:52:10.054 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:09 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:52:10.054 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:09 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:52:10.054 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:09 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:52:10.054 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:09 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:52:10.054 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:09 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:52:10.054 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:09 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:52:10.054 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:09 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:52:10.054 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:09 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:52:10.054 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:09 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:52:10.054 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:09 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:52:10.054 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:09 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:52:10.054 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:09 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:52:10.055 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:09 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:52:10.055 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:09 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:52:10.055 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:09 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:52:10.055 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:09 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:52:10.055 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:09 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:52:10.055 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:09 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:52:10.055 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:09 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:52:10.055 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:09 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:52:10.055 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:09 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:52:10.055 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:09 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:52:10.742 INFO:teuthology.orchestra.run.smithi005.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T05:52:10.742 INFO:teuthology.orchestra.run.smithi005.stdout:alertmanager.a smithi005 *:9093,9094 running (9m) 2m ago 31m 28.0M - 0.25.0 c8568f914cd2 ee99dfc435e8 2024-03-22T05:52:10.742 INFO:teuthology.orchestra.run.smithi005.stdout:grafana.a smithi186 *:3000 running (9m) 36s ago 31m 81.3M - 9.4.12 2bacad6d85d8 16fd57dbbd71 2024-03-22T05:52:10.742 INFO:teuthology.orchestra.run.smithi005.stdout:iscsi.foo.smithi005.mcdwbp smithi005 unknown 2m ago 30m 64.6M - 3.5 e1d6a67b021e 5215e03bf81b 2024-03-22T05:52:10.742 INFO:teuthology.orchestra.run.smithi005.stdout:mgr.x smithi186 *:8443,9283,8765 running (10m) 36s ago 36m 454M - 19.0.0-2299-gea01217f aa461e7c5e7f c556421a73f3 2024-03-22T05:52:10.742 INFO:teuthology.orchestra.run.smithi005.stdout:mgr.y smithi005 *:8443,9283,8765 running (20m) 2m ago 37m 511M - 19.0.0-2299-gea01217f aa461e7c5e7f 62a813eebc0d 2024-03-22T05:52:10.742 INFO:teuthology.orchestra.run.smithi005.stdout:mon.a smithi005 running (7m) 2m ago 37m 55.3M 2048M 19.0.0-2299-gea01217f aa461e7c5e7f 111a2aa24dd9 2024-03-22T05:52:10.742 INFO:teuthology.orchestra.run.smithi005.stdout:mon.b smithi186 running (8m) 36s ago 36m 53.1M 2048M 19.0.0-2299-gea01217f aa461e7c5e7f c1798feac15b 2024-03-22T05:52:10.742 INFO:teuthology.orchestra.run.smithi005.stdout:mon.c smithi005 running (7m) 2m ago 36m 42.0M 2048M 19.0.0-2299-gea01217f aa461e7c5e7f 708c72ff74f5 2024-03-22T05:52:10.742 INFO:teuthology.orchestra.run.smithi005.stdout:node-exporter.a smithi005 *:9100 running (10m) 2m ago 32m 16.8M - 1.5.0 0da6a335fe13 f635cd72b9d3 2024-03-22T05:52:10.742 INFO:teuthology.orchestra.run.smithi005.stdout:node-exporter.b smithi186 *:9100 running (10m) 36s ago 32m 14.7M - 1.5.0 0da6a335fe13 d7e0b3d20191 2024-03-22T05:52:10.743 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0 smithi005 running (5m) 2m ago 35m 44.8M 2993M 19.0.0-2299-gea01217f aa461e7c5e7f 31b6cdee5f5b 2024-03-22T05:52:10.743 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1 smithi005 running (4m) 2m ago 35m 44.2M 2993M 19.0.0-2299-gea01217f aa461e7c5e7f 7d21bc9fd3c2 2024-03-22T05:52:10.743 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2 smithi005 running (3m) 2m ago 34m 42.0M 2993M 19.0.0-2299-gea01217f aa461e7c5e7f 628204cb7c98 2024-03-22T05:52:10.743 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3 smithi005 running (2m) 2m ago 34m 14.9M 2993M 19.0.0-2299-gea01217f aa461e7c5e7f d31a01a0230c 2024-03-22T05:52:10.743 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4 smithi186 running (2m) 36s ago 34m 57.4M 3248M 19.0.0-2299-gea01217f aa461e7c5e7f 8f7619b726f3 2024-03-22T05:52:10.743 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5 smithi186 running (99s) 36s ago 33m 52.2M 3248M 19.0.0-2299-gea01217f aa461e7c5e7f b80f7eceb2d7 2024-03-22T05:52:10.743 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6 smithi186 running (62s) 36s ago 33m 51.7M 3248M 19.0.0-2299-gea01217f aa461e7c5e7f db7c81962c65 2024-03-22T05:52:10.743 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7 smithi186 running (40s) 36s ago 33m 15.1M 3248M 19.0.0-2299-gea01217f aa461e7c5e7f fcb46ba843ae 2024-03-22T05:52:10.743 INFO:teuthology.orchestra.run.smithi005.stdout:prometheus.a smithi186 *:9095 running (10m) 36s ago 32m 72.0M - 2.43.0 a07b618ecd1d a5a043588d6d 2024-03-22T05:52:10.743 INFO:teuthology.orchestra.run.smithi005.stdout:rgw.foo.smithi005.exdhqu smithi005 *:8000 running (30m) 2m ago 30m 81.5M - 17.2.0 e1d6a67b021e 6bb27fe11553 2024-03-22T05:52:10.743 INFO:teuthology.orchestra.run.smithi005.stdout:rgw.foo.smithi186.mcgodk smithi186 *:8000 running (30m) 36s ago 30m 80.2M - 17.2.0 e1d6a67b021e 19b5c20a34f7 2024-03-22T05:52:11.165 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-03-22T05:52:11.165 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": { 2024-03-22T05:52:11.165 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev)": 3 2024-03-22T05:52:11.166 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:52:11.166 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr": { 2024-03-22T05:52:11.166 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev)": 2 2024-03-22T05:52:11.166 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:52:11.166 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": { 2024-03-22T05:52:11.166 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev)": 8 2024-03-22T05:52:11.166 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:52:11.166 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw": { 2024-03-22T05:52:11.166 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-22T05:52:11.166 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:52:11.166 INFO:teuthology.orchestra.run.smithi005.stdout: "overall": { 2024-03-22T05:52:11.166 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-03-22T05:52:11.166 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev)": 13 2024-03-22T05:52:11.167 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-03-22T05:52:11.167 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-03-22T05:52:11.519 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-03-22T05:52:11.519 INFO:teuthology.orchestra.run.smithi005.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37", 2024-03-22T05:52:11.519 INFO:teuthology.orchestra.run.smithi005.stdout: "in_progress": true, 2024-03-22T05:52:11.519 INFO:teuthology.orchestra.run.smithi005.stdout: "which": "Upgrading daemons in service(s) rgw.foo", 2024-03-22T05:52:11.519 INFO:teuthology.orchestra.run.smithi005.stdout: "services_complete": [], 2024-03-22T05:52:11.519 INFO:teuthology.orchestra.run.smithi005.stdout: "progress": "", 2024-03-22T05:52:11.519 INFO:teuthology.orchestra.run.smithi005.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37 image", 2024-03-22T05:52:11.519 INFO:teuthology.orchestra.run.smithi005.stdout: "is_paused": false 2024-03-22T05:52:11.519 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-03-22T05:52:11.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:11 smithi005 ceph-mon[89904]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:52:11.703 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:52:11.703 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:52:11.703 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:52:11.703 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:52:11.703 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:52:11.703 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:52:11.703 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:52:11.703 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:52:11.703 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:52:11.703 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:52:11.703 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:52:11.703 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:52:11.703 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:52:11.704 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:52:11.704 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:52:11.704 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:52:11.704 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:52:11.704 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:52:11.704 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:52:11.704 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:52:11.704 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:52:11.704 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:52:11.704 INFO:journalctl@ceph.mon.c.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:52:11.704 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:52:11.704 INFO:journalctl@ceph.mon.c.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:52:11.704 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:52:11.704 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:52:11.704 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:52:11.705 INFO:journalctl@ceph.mon.c.smithi005.stdout: return future.result(timeout) 2024-03-22T05:52:11.705 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:52:11.705 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.__get_result() 2024-03-22T05:52:11.705 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:52:11.705 INFO:journalctl@ceph.mon.c.smithi005.stdout: raise self._exception 2024-03-22T05:52:11.705 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:52:11.705 INFO:journalctl@ceph.mon.c.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:52:11.705 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:52:11.705 INFO:journalctl@ceph.mon.c.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:52:11.705 INFO:journalctl@ceph.mon.c.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:52:11.705 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:52:11.705 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:52:11.705 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:52:11.705 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:52:11.705 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:52:11.706 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:52:11.706 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:52:11.706 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:52:11.706 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:52:11.706 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:52:11.706 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:52:11.706 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:52:11.706 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:52:11.706 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:52:11.706 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:52:11.706 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:11 smithi005 ceph-mon[84705]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:52:11.707 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:52:11.707 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:52:11.707 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:52:11.707 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:52:11.707 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:52:11.707 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:52:11.707 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:52:11.707 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:52:11.707 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:52:11.707 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:52:11.707 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:52:11.707 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:52:11.707 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:52:11.708 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:52:11.708 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:52:11.708 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:52:11.708 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:52:11.708 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:52:11.708 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:52:11.708 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:52:11.708 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:52:11.708 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:52:11.708 INFO:journalctl@ceph.mon.a.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:52:11.708 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:52:11.708 INFO:journalctl@ceph.mon.a.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:52:11.708 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:52:11.708 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:52:11.708 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:52:11.709 INFO:journalctl@ceph.mon.a.smithi005.stdout: return future.result(timeout) 2024-03-22T05:52:11.709 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:52:11.709 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.__get_result() 2024-03-22T05:52:11.709 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:52:11.709 INFO:journalctl@ceph.mon.a.smithi005.stdout: raise self._exception 2024-03-22T05:52:11.709 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:52:11.709 INFO:journalctl@ceph.mon.a.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:52:11.709 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:52:11.709 INFO:journalctl@ceph.mon.a.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:52:11.709 INFO:journalctl@ceph.mon.a.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:52:11.709 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:52:11.709 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:52:11.709 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:52:11.709 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:52:11.710 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:52:11.710 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:52:11.710 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:52:11.710 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:52:11.710 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:52:11.710 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:52:11.710 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:52:11.710 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:52:11.710 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:52:11.710 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:52:11.710 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:52:11.711 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:52:11.711 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:52:11.711 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:52:11.711 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:52:11.711 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:52:11.711 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:11 smithi005 ceph-mon[89904]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37 2024-03-22T05:52:11.711 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:11 smithi005 ceph-mon[89904]: from='client.64591 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:52:11.711 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:11 smithi005 ceph-mon[89904]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:52:11.711 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:11 smithi005 ceph-mon[89904]: from='client.? 172.21.15.5:0/2496093144' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:52:11.711 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:52:11.711 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:52:11.711 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:52:11.712 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:52:11.712 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:52:11.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:11 smithi005 ceph-mon[84705]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37 2024-03-22T05:52:11.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:11 smithi005 ceph-mon[84705]: from='client.64591 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:52:11.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:11 smithi005 ceph-mon[84705]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:52:11.712 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:11 smithi005 ceph-mon[84705]: from='client.? 172.21.15.5:0/2496093144' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:52:11.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:11 smithi186 ceph-mon[67903]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:52:11.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:52:11.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:52:11.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:52:11.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:52:11.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:52:11.794 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:52:11.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:52:11.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:52:11.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:52:11.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:52:11.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:52:11.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:52:11.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:52:11.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:52:11.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:52:11.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:52:11.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:52:11.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:52:11.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:52:11.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:52:11.795 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:52:11.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:52:11.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:52:11.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:52:11.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:52:11.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:52:11.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:52:11.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:52:11.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: return future.result(timeout) 2024-03-22T05:52:11.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:52:11.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.__get_result() 2024-03-22T05:52:11.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:52:11.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: raise self._exception 2024-03-22T05:52:11.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:52:11.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:52:11.796 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:52:11.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:52:11.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:52:11.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:52:11.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:52:11.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:52:11.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:52:11.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:52:11.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:52:11.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:52:11.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:52:11.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:52:11.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:52:11.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:52:11.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:52:11.797 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:52:11.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:52:11.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:52:11.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:52:11.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:52:11.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:52:11.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:52:11.798 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:52:11.798 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:11 smithi186 ceph-mon[67903]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37 2024-03-22T05:52:11.798 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:11 smithi186 ceph-mon[67903]: from='client.64591 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:52:11.798 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:11 smithi186 ceph-mon[67903]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:52:11.798 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:11 smithi186 ceph-mon[67903]: from='client.? 172.21.15.5:0/2496093144' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:52:12.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:12 smithi005 ceph-mon[84705]: from='client.64597 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:52:12.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:12 smithi005 ceph-mon[84705]: from='client.64603 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:52:12.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:12 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:12.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:12 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:52:12.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:12 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:52:12.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:12 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:12.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:12 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:52:12.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:12 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:12.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:12 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:52:12.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:12 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:12.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:12 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:52:12.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:12 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:12.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:12 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:52:12.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:12 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:12.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:12 smithi005 ceph-mon[89904]: from='client.64597 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:52:12.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:12 smithi005 ceph-mon[89904]: from='client.64603 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:52:12.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:12 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:12.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:12 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:52:12.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:12 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:52:12.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:12 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:12.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:12 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:52:12.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:12 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:12.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:12 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:52:12.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:12 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:12.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:12 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:52:12.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:12 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:12.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:12 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:52:12.704 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:12 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:12.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:12 smithi186 ceph-mon[67903]: from='client.64597 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:52:12.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:12 smithi186 ceph-mon[67903]: from='client.64603 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:52:12.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:12 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:12.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:12 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:52:12.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:12 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:52:12.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:12 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:12.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:12 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:52:12.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:12 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:12.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:12 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:52:12.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:12 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:12.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:12 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:52:12.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:12 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:12.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:12 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:52:12.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:12 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:13.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:13 smithi005 ceph-mon[84705]: from='client.64615 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:52:13.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:13 smithi005 ceph-mon[84705]: Upgrade: Target is version 19.0.0-2299-gea01217f (squid) 2024-03-22T05:52:13.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:13 smithi005 ceph-mon[84705]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, digests ['quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9'] 2024-03-22T05:52:13.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:13 smithi005 ceph-mon[84705]: Upgrade: Setting container_image for all mgr 2024-03-22T05:52:13.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:13 smithi005 ceph-mon[84705]: Upgrade: Setting container_image for all mon 2024-03-22T05:52:13.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:13 smithi005 ceph-mon[84705]: Upgrade: Setting container_image for all crash 2024-03-22T05:52:13.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:13 smithi005 ceph-mon[84705]: Upgrade: Setting container_image for all osd 2024-03-22T05:52:13.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:13 smithi005 ceph-mon[84705]: Upgrade: Setting container_image for all mds 2024-03-22T05:52:13.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:13 smithi005 ceph-mon[84705]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:52:13.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:13 smithi005 ceph-mon[89904]: from='client.64615 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:52:13.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:13 smithi005 ceph-mon[89904]: Upgrade: Target is version 19.0.0-2299-gea01217f (squid) 2024-03-22T05:52:13.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:13 smithi005 ceph-mon[89904]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, digests ['quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9'] 2024-03-22T05:52:13.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:13 smithi005 ceph-mon[89904]: Upgrade: Setting container_image for all mgr 2024-03-22T05:52:13.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:13 smithi005 ceph-mon[89904]: Upgrade: Setting container_image for all mon 2024-03-22T05:52:13.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:13 smithi005 ceph-mon[89904]: Upgrade: Setting container_image for all crash 2024-03-22T05:52:13.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:13 smithi005 ceph-mon[89904]: Upgrade: Setting container_image for all osd 2024-03-22T05:52:13.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:13 smithi005 ceph-mon[89904]: Upgrade: Setting container_image for all mds 2024-03-22T05:52:13.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:13 smithi005 ceph-mon[89904]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:52:13.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:13 smithi186 ceph-mon[67903]: from='client.64615 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:52:13.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:13 smithi186 ceph-mon[67903]: Upgrade: Target is version 19.0.0-2299-gea01217f (squid) 2024-03-22T05:52:13.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:13 smithi186 ceph-mon[67903]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, digests ['quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9'] 2024-03-22T05:52:13.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:13 smithi186 ceph-mon[67903]: Upgrade: Setting container_image for all mgr 2024-03-22T05:52:13.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:13 smithi186 ceph-mon[67903]: Upgrade: Setting container_image for all mon 2024-03-22T05:52:13.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:13 smithi186 ceph-mon[67903]: Upgrade: Setting container_image for all crash 2024-03-22T05:52:13.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:13 smithi186 ceph-mon[67903]: Upgrade: Setting container_image for all osd 2024-03-22T05:52:13.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:13 smithi186 ceph-mon[67903]: Upgrade: Setting container_image for all mds 2024-03-22T05:52:13.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:13 smithi186 ceph-mon[67903]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:52:15.197 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:14 smithi005 ceph-mon[84705]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:52:15.197 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:14 smithi005 ceph-mon[84705]: Upgrade: Updating rgw.foo.smithi005.exdhqu (1/2) 2024-03-22T05:52:15.197 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:14 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:15.198 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:14 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi005.exdhqu", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-22T05:52:15.198 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:14 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi005.exdhqu", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-22T05:52:15.198 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:14 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:52:15.198 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:14 smithi005 ceph-mon[84705]: Deploying daemon rgw.foo.smithi005.exdhqu on smithi005 2024-03-22T05:52:15.198 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:14 smithi005 ceph-mon[84705]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:52:15.198 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:14 smithi005 ceph-mon[89904]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:52:15.198 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:14 smithi005 ceph-mon[89904]: Upgrade: Updating rgw.foo.smithi005.exdhqu (1/2) 2024-03-22T05:52:15.198 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:14 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:15.198 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:14 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi005.exdhqu", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-22T05:52:15.198 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:14 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi005.exdhqu", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-22T05:52:15.198 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:14 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:52:15.199 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:14 smithi005 ceph-mon[89904]: Deploying daemon rgw.foo.smithi005.exdhqu on smithi005 2024-03-22T05:52:15.199 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:14 smithi005 ceph-mon[89904]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:52:15.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:14 smithi186 ceph-mon[67903]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:52:15.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:14 smithi186 ceph-mon[67903]: Upgrade: Updating rgw.foo.smithi005.exdhqu (1/2) 2024-03-22T05:52:15.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:14 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:15.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:14 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi005.exdhqu", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-22T05:52:15.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:14 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi005.exdhqu", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-22T05:52:15.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:14 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:52:15.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:14 smithi186 ceph-mon[67903]: Deploying daemon rgw.foo.smithi005.exdhqu on smithi005 2024-03-22T05:52:15.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:14 smithi186 ceph-mon[67903]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:52:17.181 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:16 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:52:16] "GET /metrics HTTP/1.1" 200 38368 "" "Prometheus/2.43.0" 2024-03-22T05:52:17.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:17 smithi005 ceph-mon[84705]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:52:17.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:17 smithi005 ceph-mon[89904]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:52:17.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:17 smithi186 ceph-mon[67903]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:52:19.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:19 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:19.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:19 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:19.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:19 smithi005 ceph-mon[84705]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:52:19.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:19 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:19.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:19 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:19.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:19 smithi005 ceph-mon[89904]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:52:19.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:19 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:19.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:19 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:19.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:19 smithi186 ceph-mon[67903]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:52:21.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:20 smithi186 ceph-mon[67903]: Upgrade: Updating rgw.foo.smithi186.mcgodk (2/2) 2024-03-22T05:52:21.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:20 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:21.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:20 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi186.mcgodk", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-22T05:52:21.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:20 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi186.mcgodk", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-22T05:52:21.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:20 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:52:21.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:20 smithi186 ceph-mon[67903]: Deploying daemon rgw.foo.smithi186.mcgodk on smithi186 2024-03-22T05:52:21.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:20 smithi186 ceph-mon[67903]: pgmap v298: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 85 B/s wr, 62 op/s 2024-03-22T05:52:21.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:20 smithi005 ceph-mon[84705]: Upgrade: Updating rgw.foo.smithi186.mcgodk (2/2) 2024-03-22T05:52:21.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:20 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:21.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:20 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi186.mcgodk", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-22T05:52:21.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:20 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi186.mcgodk", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-22T05:52:21.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:20 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:52:21.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:20 smithi005 ceph-mon[84705]: Deploying daemon rgw.foo.smithi186.mcgodk on smithi186 2024-03-22T05:52:21.203 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:20 smithi005 ceph-mon[84705]: pgmap v298: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 85 B/s wr, 62 op/s 2024-03-22T05:52:21.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:20 smithi005 ceph-mon[89904]: Upgrade: Updating rgw.foo.smithi186.mcgodk (2/2) 2024-03-22T05:52:21.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:20 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:21.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:20 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi186.mcgodk", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-22T05:52:21.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:20 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi186.mcgodk", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-22T05:52:21.204 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:20 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:52:21.204 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:20 smithi005 ceph-mon[89904]: Deploying daemon rgw.foo.smithi186.mcgodk on smithi186 2024-03-22T05:52:21.204 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:20 smithi005 ceph-mon[89904]: pgmap v298: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 85 B/s wr, 62 op/s 2024-03-22T05:52:23.498 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:23 smithi186 ceph-mon[67903]: pgmap v299: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 85 B/s wr, 124 op/s 2024-03-22T05:52:23.498 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:23 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:52:23.665 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:23 smithi005 ceph-mon[84705]: pgmap v299: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 85 B/s wr, 124 op/s 2024-03-22T05:52:23.665 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:23 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:52:23.666 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:23 smithi005 ceph-mon[89904]: pgmap v299: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 85 B/s wr, 124 op/s 2024-03-22T05:52:23.666 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:23 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:52:25.270 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:25 smithi186 ceph-mon[67903]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:52:25.270 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:25 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:25.270 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:25 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:25.270 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:25 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:52:25.270 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:25 smithi186 ceph-mon[67903]: pgmap v300: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 85 B/s wr, 124 op/s 2024-03-22T05:52:25.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:25 smithi005 ceph-mon[84705]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:52:25.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:25 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:25.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:25 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:25.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:25 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:52:25.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:25 smithi005 ceph-mon[84705]: pgmap v300: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 85 B/s wr, 124 op/s 2024-03-22T05:52:25.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:25 smithi005 ceph-mon[89904]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:52:25.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:25 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:25.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:25 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:25.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:25 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:52:25.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:25 smithi005 ceph-mon[89904]: pgmap v300: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 85 B/s wr, 124 op/s 2024-03-22T05:52:27.340 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:26 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:52:26] "GET /metrics HTTP/1.1" 200 38360 "" "Prometheus/2.43.0" 2024-03-22T05:52:27.340 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:27 smithi005 ceph-mon[84705]: pgmap v301: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 101 KiB/s rd, 170 B/s wr, 158 op/s 2024-03-22T05:52:27.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:27 smithi186 ceph-mon[67903]: pgmap v301: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 101 KiB/s rd, 170 B/s wr, 158 op/s 2024-03-22T05:52:27.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:27 smithi005 ceph-mon[89904]: pgmap v301: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 101 KiB/s rd, 170 B/s wr, 158 op/s 2024-03-22T05:52:28.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:28 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:28.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:28 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:28.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:28 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:28.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:28 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:28.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:28 smithi005 ceph-mon[89904]: pgmap v302: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 130 KiB/s rd, 170 B/s wr, 205 op/s 2024-03-22T05:52:28.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:28 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:28.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:28 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:28.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:28 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:28.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:28 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:28.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:28 smithi005 ceph-mon[84705]: pgmap v302: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 130 KiB/s rd, 170 B/s wr, 205 op/s 2024-03-22T05:52:29.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:28 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:29.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:28 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:29.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:28 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:29.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:28 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:29.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:28 smithi186 ceph-mon[67903]: pgmap v302: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 130 KiB/s rd, 170 B/s wr, 205 op/s 2024-03-22T05:52:29.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:29 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:29.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:29 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:29.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:29 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:29.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:29 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:29.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:29 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:29.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:29 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:29.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:29 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:29.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:29 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:30.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:29 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:30.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:29 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:30.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:29 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:30.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:29 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:30.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:30 smithi005 ceph-mon[89904]: pgmap v303: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 157 KiB/s rd, 170 B/s wr, 248 op/s 2024-03-22T05:52:30.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:30 smithi005 ceph-mon[84705]: pgmap v303: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 157 KiB/s rd, 170 B/s wr, 248 op/s 2024-03-22T05:52:31.043 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:30 smithi186 ceph-mon[67903]: pgmap v303: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 157 KiB/s rd, 170 B/s wr, 248 op/s 2024-03-22T05:52:33.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:33 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:33.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:33 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:33.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:33 smithi005 ceph-mon[89904]: pgmap v304: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 118 KiB/s rd, 85 B/s wr, 187 op/s 2024-03-22T05:52:33.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:33 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:33.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:33 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:33.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:33 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:52:33.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:33 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:52:33.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:33 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:33.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:33 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:52:33.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:33 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:52:33.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:33 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:52:33.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:33 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:33.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:33 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:33.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:33 smithi005 ceph-mon[84705]: pgmap v304: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 118 KiB/s rd, 85 B/s wr, 187 op/s 2024-03-22T05:52:33.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:33 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:33.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:33 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:33.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:33 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:52:33.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:33 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:52:33.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:33 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:33.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:33 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:52:33.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:33 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:52:33.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:33 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:52:33.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:33 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:33.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:33 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:33.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:33 smithi186 ceph-mon[67903]: pgmap v304: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 118 KiB/s rd, 85 B/s wr, 187 op/s 2024-03-22T05:52:33.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:33 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:33.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:33 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:33.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:33 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:52:33.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:33 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:52:33.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:33 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:33.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:33 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:52:33.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:33 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:52:33.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:33 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:52:34.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:34 smithi005 ceph-mon[89904]: pgmap v305: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 91 KiB/s rd, 97 B/s wr, 142 op/s 2024-03-22T05:52:34.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:34 smithi005 ceph-mon[89904]: pgmap v306: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 112 KiB/s rd, 121 B/s wr, 176 op/s 2024-03-22T05:52:34.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:34 smithi005 ceph-mon[89904]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:52:34.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:34 smithi005 ceph-mon[89904]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:52:34.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:34 smithi005 ceph-mon[84705]: pgmap v305: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 91 KiB/s rd, 97 B/s wr, 142 op/s 2024-03-22T05:52:34.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:34 smithi005 ceph-mon[84705]: pgmap v306: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 112 KiB/s rd, 121 B/s wr, 176 op/s 2024-03-22T05:52:34.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:34 smithi005 ceph-mon[84705]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:52:34.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:34 smithi005 ceph-mon[84705]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:52:34.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:34 smithi186 ceph-mon[67903]: pgmap v305: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 91 KiB/s rd, 97 B/s wr, 142 op/s 2024-03-22T05:52:34.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:34 smithi186 ceph-mon[67903]: pgmap v306: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 112 KiB/s rd, 121 B/s wr, 176 op/s 2024-03-22T05:52:34.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:34 smithi186 ceph-mon[67903]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:52:34.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:34 smithi186 ceph-mon[67903]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:52:34.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:52:34.664+0000 7f7f0ecc9700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:52:34.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:52:34.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:52:34.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:52:34.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:52:34.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:52:34.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:52:34.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:52:34.954 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:52:34.954 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:52:34.954 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:52:34.954 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:52:34.954 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:52:34.954 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:52:34.954 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:52:34.954 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:52:34.954 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:52:34.954 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:52:34.954 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:52:34.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:52:34.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:52:34.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:52:34.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:52:34.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:52:34.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:52:34.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:52:34.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:52:34.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:52:34.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:52:34.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return future.result(timeout) 2024-03-22T05:52:34.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:52:34.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.__get_result() 2024-03-22T05:52:34.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:52:34.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: raise self._exception 2024-03-22T05:52:34.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:52:34.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: config_blobs=daemon_spec.final_config, 2024-03-22T05:52:34.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:52:34.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:52:34.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:52:34.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:52:34.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:52:34.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:52:34.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:52:34.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:52:34.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:52:34.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:52:34.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:52:34.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:52:34.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:52:34.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:52:34.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:52:34.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:52:34.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:52:34.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:52:34.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:52:34.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:52:34.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:52:34.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:52:34.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:34 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:52:35.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:52:35.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:52:35.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:52:35.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:52:35.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:52:35.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:52:35.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:52:35.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:52:35.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:35.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi005.exdhqu"}]: dispatch 2024-03-22T05:52:35.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi005.exdhqu"}]: dispatch 2024-03-22T05:52:35.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi005.exdhqu"}]': finished 2024-03-22T05:52:35.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi186.mcgodk"}]: dispatch 2024-03-22T05:52:35.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi186.mcgodk"}]: dispatch 2024-03-22T05:52:35.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi186.mcgodk"}]': finished 2024-03-22T05:52:35.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:52:35.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:35.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:52:35.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:52:35.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:35.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:52:35.455 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:35.455 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:52:35.455 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:35.455 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:52:35.455 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:52:35.455 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:52:35.455 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:52:35.455 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:52:35.455 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:52:35.456 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:52:35.456 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:52:35.456 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:52:35.456 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:52:35.456 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:52:35.456 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:52:35.456 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:35.456 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi005.exdhqu"}]: dispatch 2024-03-22T05:52:35.456 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi005.exdhqu"}]: dispatch 2024-03-22T05:52:35.456 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi005.exdhqu"}]': finished 2024-03-22T05:52:35.456 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi186.mcgodk"}]: dispatch 2024-03-22T05:52:35.456 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi186.mcgodk"}]: dispatch 2024-03-22T05:52:35.456 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi186.mcgodk"}]': finished 2024-03-22T05:52:35.457 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:52:35.457 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:35.457 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:52:35.457 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:52:35.457 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:35.457 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:52:35.457 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:35.457 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:52:35.457 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:35.457 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:52:35.457 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:52:35.457 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:52:35.457 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:52:35.458 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:52:35.458 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:52:35.458 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-22T05:52:35.458 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-22T05:52:35.458 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-22T05:52:35.458 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:52:35.458 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:52:35.458 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-22T05:52:35.458 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-22T05:52:35.459 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-22T05:52:35.459 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-22T05:52:35.459 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-22T05:52:35.459 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-22T05:52:35.459 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-03-22T05:52:35.459 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-22T05:52:35.459 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-22T05:52:35.459 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-22T05:52:35.459 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-22T05:52:35.459 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-22T05:52:35.459 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-03-22T05:52:35.459 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-22T05:52:35.459 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-22T05:52:35.460 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-22T05:52:35.460 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:52:35.460 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:52:35.460 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:52:35.460 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:52:35.460 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-22T05:52:35.460 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-22T05:52:35.460 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-22T05:52:35.460 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:52:35.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:52:35.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-22T05:52:35.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-22T05:52:35.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-22T05:52:35.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-22T05:52:35.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-22T05:52:35.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-22T05:52:35.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-03-22T05:52:35.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-22T05:52:35.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-22T05:52:35.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-22T05:52:35.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-22T05:52:35.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-22T05:52:35.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-03-22T05:52:35.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-22T05:52:35.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-22T05:52:35.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-22T05:52:35.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:52:35.462 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:52:35.463 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-22T05:52:35.463 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-22T05:52:35.463 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-22T05:52:35.463 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-22T05:52:35.463 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-22T05:52:35.463 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-22T05:52:35.463 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-22T05:52:35.463 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-22T05:52:35.463 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-22T05:52:35.464 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-22T05:52:35.464 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-22T05:52:35.464 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:52:35.464 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:52:35.464 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:52:35.464 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:52:35.464 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:52:35.464 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:52:35.464 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:52:35.464 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:52:35.464 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:52:35.464 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:52:35.464 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:52:35.464 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:52:35.465 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-22T05:52:35.465 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-22T05:52:35.465 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-22T05:52:35.465 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:52:35.465 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:52:35.465 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:52:35.465 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:35.465 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:52:35.465 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:52:35.465 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:52:35.466 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-22T05:52:35.466 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-22T05:52:35.466 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-22T05:52:35.466 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-22T05:52:35.466 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-22T05:52:35.466 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-22T05:52:35.466 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-22T05:52:35.466 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-22T05:52:35.466 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-22T05:52:35.466 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-22T05:52:35.466 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-22T05:52:35.467 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:52:35.467 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:52:35.467 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:52:35.467 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:52:35.467 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:52:35.467 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:52:35.467 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:52:35.467 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:52:35.467 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:52:35.467 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:52:35.467 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:52:35.467 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:52:35.467 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-22T05:52:35.468 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-22T05:52:35.468 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-22T05:52:35.468 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:52:35.468 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:52:35.468 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:52:35.468 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:35.468 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:52:35.468 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:52:35.468 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:35 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:52:35.546 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:52:35.547 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:52:35.547 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:52:35.547 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:52:35.547 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:52:35.547 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:52:35.547 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:52:35.547 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:52:35.547 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:35.547 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi005.exdhqu"}]: dispatch 2024-03-22T05:52:35.547 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi005.exdhqu"}]: dispatch 2024-03-22T05:52:35.547 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi005.exdhqu"}]': finished 2024-03-22T05:52:35.547 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi186.mcgodk"}]: dispatch 2024-03-22T05:52:35.547 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi186.mcgodk"}]: dispatch 2024-03-22T05:52:35.547 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi186.mcgodk"}]': finished 2024-03-22T05:52:35.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:52:35.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:35.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:52:35.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:52:35.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:35.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:52:35.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:35.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:52:35.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:35.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:52:35.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:52:35.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:52:35.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:52:35.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:52:35.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:52:35.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-22T05:52:35.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-22T05:52:35.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-22T05:52:35.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:52:35.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:52:35.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-22T05:52:35.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-22T05:52:35.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-22T05:52:35.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-22T05:52:35.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-22T05:52:35.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-22T05:52:35.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-03-22T05:52:35.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-22T05:52:35.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-22T05:52:35.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-22T05:52:35.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-22T05:52:35.550 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-22T05:52:35.550 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-03-22T05:52:35.550 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-22T05:52:35.550 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-22T05:52:35.550 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-22T05:52:35.550 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:52:35.550 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:52:35.550 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-22T05:52:35.550 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-22T05:52:35.550 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-22T05:52:35.550 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-22T05:52:35.550 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-22T05:52:35.550 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-22T05:52:35.550 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-22T05:52:35.551 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-22T05:52:35.551 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-22T05:52:35.551 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-22T05:52:35.551 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-22T05:52:35.551 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:52:35.551 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:52:35.551 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:52:35.551 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:52:35.551 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:52:35.551 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:52:35.551 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:52:35.551 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:52:35.551 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:52:35.551 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:52:35.551 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:52:35.551 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-22T05:52:35.552 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-22T05:52:35.552 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-22T05:52:35.552 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-22T05:52:35.552 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:52:35.552 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:52:35.552 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:52:35.552 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:35.552 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:52:35.552 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:52:35.552 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:35 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:52:36.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-mon[89904]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:52:36.453 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:52:36.453 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:52:36.453 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:52:36.453 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:52:36.453 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:52:36.453 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:52:36.453 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:52:36.453 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:52:36.453 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:52:36.453 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:52:36.453 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:52:36.453 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:52:36.454 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:52:36.454 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:52:36.454 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:52:36.454 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:52:36.454 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:52:36.454 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:52:36.454 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:52:36.454 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:52:36.454 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:52:36.454 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:52:36.454 INFO:journalctl@ceph.mon.c.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:52:36.454 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:52:36.454 INFO:journalctl@ceph.mon.c.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:52:36.454 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:52:36.455 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:52:36.455 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:52:36.455 INFO:journalctl@ceph.mon.c.smithi005.stdout: return future.result(timeout) 2024-03-22T05:52:36.455 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:52:36.455 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.__get_result() 2024-03-22T05:52:36.455 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:52:36.455 INFO:journalctl@ceph.mon.c.smithi005.stdout: raise self._exception 2024-03-22T05:52:36.455 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:52:36.455 INFO:journalctl@ceph.mon.c.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:52:36.455 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:52:36.455 INFO:journalctl@ceph.mon.c.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:52:36.455 INFO:journalctl@ceph.mon.c.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:52:36.455 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:52:36.455 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:52:36.456 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:52:36.456 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:52:36.456 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:52:36.456 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:52:36.456 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:52:36.456 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:52:36.456 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:52:36.456 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:52:36.456 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:52:36.456 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:52:36.456 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:52:36.456 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:52:36.456 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:52:36.457 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-mon[84705]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:52:36.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:52:36.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:52:36.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:52:36.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:52:36.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:52:36.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:52:36.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:52:36.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:52:36.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:52:36.458 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:52:36.458 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:52:36.458 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:52:36.458 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:52:36.458 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:52:36.458 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:52:36.458 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:52:36.458 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:52:36.458 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:52:36.458 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:52:36.458 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:52:36.458 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:52:36.458 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:52:36.458 INFO:journalctl@ceph.mon.a.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:52:36.459 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:52:36.459 INFO:journalctl@ceph.mon.a.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:52:36.459 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:52:36.459 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:52:36.459 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:52:36.459 INFO:journalctl@ceph.mon.a.smithi005.stdout: return future.result(timeout) 2024-03-22T05:52:36.459 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:52:36.459 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.__get_result() 2024-03-22T05:52:36.459 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:52:36.459 INFO:journalctl@ceph.mon.a.smithi005.stdout: raise self._exception 2024-03-22T05:52:36.460 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:52:36.460 INFO:journalctl@ceph.mon.a.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:52:36.460 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:52:36.460 INFO:journalctl@ceph.mon.a.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:52:36.460 INFO:journalctl@ceph.mon.a.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:52:36.460 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:52:36.460 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:52:36.460 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:52:36.460 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:52:36.460 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:52:36.460 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:52:36.460 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:52:36.460 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:52:36.461 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:52:36.461 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:52:36.461 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:52:36.461 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:52:36.461 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:52:36.461 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:52:36.461 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:52:36.461 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:52:36.461 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:52:36.461 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:52:36.462 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:52:36.462 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:52:36.462 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-mon[89904]: Upgrade: Setting container_image for all rgw 2024-03-22T05:52:36.462 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-mon[89904]: Upgrade: Setting container_image for all rbd-mirror 2024-03-22T05:52:36.462 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-mon[89904]: Upgrade: Setting container_image for all ceph-exporter 2024-03-22T05:52:36.462 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-mon[89904]: Upgrade: Setting container_image for all nfs 2024-03-22T05:52:36.462 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-mon[89904]: Upgrade: Setting container_image for all nvmeof 2024-03-22T05:52:36.462 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-mon[89904]: Upgrade: Finalizing container_image settings 2024-03-22T05:52:36.462 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-mon[89904]: pgmap v307: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 81 KiB/s rd, 0 B/s wr, 128 op/s 2024-03-22T05:52:36.462 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-mon[89904]: Upgrade: Complete! 2024-03-22T05:52:36.462 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-mon[89904]: pgmap v308: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 0 B/s wr, 81 op/s 2024-03-22T05:52:36.463 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-mon[89904]: pgmap v309: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:52:36.463 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-mon[89904]: Checking dashboard <-> RGW credentials 2024-03-22T05:52:36.463 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-mon[89904]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:52:36.463 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-mon[89904]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:52:36.463 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:52:36.463 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:52:36.463 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:52:36.463 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:52:36.463 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:52:36.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-mon[84705]: Upgrade: Setting container_image for all rgw 2024-03-22T05:52:36.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-mon[84705]: Upgrade: Setting container_image for all rbd-mirror 2024-03-22T05:52:36.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-mon[84705]: Upgrade: Setting container_image for all ceph-exporter 2024-03-22T05:52:36.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-mon[84705]: Upgrade: Setting container_image for all nfs 2024-03-22T05:52:36.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-mon[84705]: Upgrade: Setting container_image for all nvmeof 2024-03-22T05:52:36.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-mon[84705]: Upgrade: Finalizing container_image settings 2024-03-22T05:52:36.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-mon[84705]: pgmap v307: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 81 KiB/s rd, 0 B/s wr, 128 op/s 2024-03-22T05:52:36.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-mon[84705]: Upgrade: Complete! 2024-03-22T05:52:36.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-mon[84705]: pgmap v308: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 0 B/s wr, 81 op/s 2024-03-22T05:52:36.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-mon[84705]: pgmap v309: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:52:36.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-mon[84705]: Checking dashboard <-> RGW credentials 2024-03-22T05:52:36.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-mon[84705]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:52:36.464 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-mon[84705]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:52:36.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:36 smithi186 ceph-mon[67903]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:52:36.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:52:36.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:52:36.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:52:36.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:52:36.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:52:36.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:52:36.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:52:36.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:52:36.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:52:36.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:52:36.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:52:36.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:52:36.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:52:36.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:52:36.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:52:36.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:52:36.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:52:36.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:52:36.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:52:36.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:52:36.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:52:36.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:52:36.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:52:36.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:52:36.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:52:36.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:52:36.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:52:36.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:52:36.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: return future.result(timeout) 2024-03-22T05:52:36.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:52:36.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.__get_result() 2024-03-22T05:52:36.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:52:36.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: raise self._exception 2024-03-22T05:52:36.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:52:36.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:52:36.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:52:36.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:52:36.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:52:36.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:52:36.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:52:36.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:52:36.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:52:36.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:52:36.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:52:36.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:52:36.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:52:36.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:52:36.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:52:36.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:52:36.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:52:36.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:52:36.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:52:36.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:52:36.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:52:36.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:52:36.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:52:36.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:52:36.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:52:36.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:36 smithi186 ceph-mon[67903]: Upgrade: Setting container_image for all rgw 2024-03-22T05:52:36.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:36 smithi186 ceph-mon[67903]: Upgrade: Setting container_image for all rbd-mirror 2024-03-22T05:52:36.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:36 smithi186 ceph-mon[67903]: Upgrade: Setting container_image for all ceph-exporter 2024-03-22T05:52:36.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:36 smithi186 ceph-mon[67903]: Upgrade: Setting container_image for all nfs 2024-03-22T05:52:36.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:36 smithi186 ceph-mon[67903]: Upgrade: Setting container_image for all nvmeof 2024-03-22T05:52:36.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:36 smithi186 ceph-mon[67903]: Upgrade: Finalizing container_image settings 2024-03-22T05:52:36.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:36 smithi186 ceph-mon[67903]: pgmap v307: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 81 KiB/s rd, 0 B/s wr, 128 op/s 2024-03-22T05:52:36.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:36 smithi186 ceph-mon[67903]: Upgrade: Complete! 2024-03-22T05:52:36.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:36 smithi186 ceph-mon[67903]: pgmap v308: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 0 B/s wr, 81 op/s 2024-03-22T05:52:36.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:36 smithi186 ceph-mon[67903]: pgmap v309: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:52:36.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:36 smithi186 ceph-mon[67903]: Checking dashboard <-> RGW credentials 2024-03-22T05:52:36.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:36 smithi186 ceph-mon[67903]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:52:36.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:36 smithi186 ceph-mon[67903]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:52:36.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:52:36.794+0000 7f7f0ecc9700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:52:36.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:52:36.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:52:36.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:52:36.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:52:36.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:52:36.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:52:36.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:52:36.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:52:36.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:52:36.954 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:52:36.954 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:52:36.954 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:52:36.954 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:52:36.954 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:52:36.954 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:52:36.954 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:52:36.954 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:52:36.954 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:52:36.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:52:36.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:52:36.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:52:36.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:52:36.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:52:36.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:52:36.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:52:36.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:52:36.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:52:36.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:52:36.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return future.result(timeout) 2024-03-22T05:52:36.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:52:36.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.__get_result() 2024-03-22T05:52:36.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:52:36.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: raise self._exception 2024-03-22T05:52:36.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:52:36.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: config_blobs=daemon_spec.final_config, 2024-03-22T05:52:36.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:52:36.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:52:36.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:52:36.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:52:36.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:52:36.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:52:36.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:52:36.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:52:36.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:52:36.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:52:36.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:52:36.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:52:36.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:52:36.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:52:36.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:52:36.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:52:36.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:52:36.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:52:36.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:52:36.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:52:36.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:52:36.958 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:52:36.958 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:52:37.452 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:36 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:52:36] "GET /metrics HTTP/1.1" 200 38360 "" "Prometheus/2.43.0" 2024-03-22T05:52:37.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:37 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:52:37.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:37 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:52:37.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:37 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:52:37.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:37 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:37.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:37 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:52:37.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:37 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:52:37.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:37 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:52:37.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:37 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:52:37.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:37 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:52:37.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:37 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:52:37.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:37 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:37.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:37 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:52:37.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:37 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:52:37.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:37 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:52:37.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:37 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:52:37.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:37 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:52:37.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:37 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:52:37.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:37 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:37.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:37 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:52:37.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:37 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:52:37.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:37 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:52:38.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:38 smithi005 ceph-mon[89904]: pgmap v310: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 2 op/s 2024-03-22T05:52:38.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:38 smithi005 ceph-mon[89904]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:52:38.453 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:52:38.453 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:52:38.453 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:52:38.453 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:52:38.453 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:52:38.453 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:52:38.453 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:52:38.453 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:52:38.453 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:52:38.454 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:52:38.454 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:52:38.454 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:52:38.454 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:52:38.454 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:52:38.454 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:52:38.454 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:52:38.454 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:52:38.454 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:52:38.454 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:52:38.454 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:52:38.454 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:52:38.454 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:52:38.455 INFO:journalctl@ceph.mon.c.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:52:38.455 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:52:38.455 INFO:journalctl@ceph.mon.c.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:52:38.455 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:52:38.455 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:52:38.455 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:52:38.455 INFO:journalctl@ceph.mon.c.smithi005.stdout: return future.result(timeout) 2024-03-22T05:52:38.455 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:52:38.455 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.__get_result() 2024-03-22T05:52:38.455 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:52:38.455 INFO:journalctl@ceph.mon.c.smithi005.stdout: raise self._exception 2024-03-22T05:52:38.455 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:52:38.455 INFO:journalctl@ceph.mon.c.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:52:38.455 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:52:38.455 INFO:journalctl@ceph.mon.c.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:52:38.455 INFO:journalctl@ceph.mon.c.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:52:38.456 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:52:38.456 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:52:38.456 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:52:38.456 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:52:38.456 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:52:38.456 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:52:38.456 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:52:38.456 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:52:38.456 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:52:38.456 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:52:38.456 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:52:38.456 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:52:38.456 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:52:38.456 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:52:38.456 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:52:38.457 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:38 smithi005 ceph-mon[84705]: pgmap v310: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 2 op/s 2024-03-22T05:52:38.457 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:38 smithi005 ceph-mon[84705]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:52:38.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:52:38.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:52:38.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:52:38.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:52:38.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:52:38.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:52:38.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:52:38.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:52:38.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:52:38.458 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:52:38.458 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:52:38.458 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:52:38.458 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:52:38.458 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:52:38.458 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:52:38.458 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:52:38.458 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:52:38.458 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:52:38.458 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:52:38.458 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:52:38.458 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:52:38.458 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:52:38.458 INFO:journalctl@ceph.mon.a.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:52:38.458 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:52:38.459 INFO:journalctl@ceph.mon.a.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:52:38.459 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:52:38.459 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:52:38.459 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:52:38.459 INFO:journalctl@ceph.mon.a.smithi005.stdout: return future.result(timeout) 2024-03-22T05:52:38.459 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:52:38.459 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.__get_result() 2024-03-22T05:52:38.459 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:52:38.459 INFO:journalctl@ceph.mon.a.smithi005.stdout: raise self._exception 2024-03-22T05:52:38.459 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:52:38.459 INFO:journalctl@ceph.mon.a.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:52:38.459 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:52:38.459 INFO:journalctl@ceph.mon.a.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:52:38.459 INFO:journalctl@ceph.mon.a.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:52:38.459 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:52:38.459 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:52:38.459 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:52:38.460 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:52:38.460 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:52:38.460 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:52:38.460 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:52:38.460 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:52:38.460 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:52:38.460 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:52:38.460 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:52:38.460 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:52:38.460 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:52:38.460 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:52:38.460 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:52:38.460 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:52:38.460 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:52:38.460 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:52:38.461 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:52:38.461 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:52:38.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:38 smithi005 ceph-mon[84705]: pgmap v311: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 2 op/s 2024-03-22T05:52:38.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:38 smithi005 ceph-mon[84705]: pgmap v312: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 2 op/s 2024-03-22T05:52:38.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:38 smithi005 ceph-mon[84705]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:52:38.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:38 smithi005 ceph-mon[84705]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:52:38.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:38 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:52:38.461 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:38 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:38.461 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:52:38.461 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:52:38.461 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:52:38.461 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:52:38.461 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:52:38.462 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:38 smithi005 ceph-mon[89904]: pgmap v311: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 2 op/s 2024-03-22T05:52:38.462 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:38 smithi005 ceph-mon[89904]: pgmap v312: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 2 op/s 2024-03-22T05:52:38.462 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:38 smithi005 ceph-mon[89904]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:52:38.462 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:38 smithi005 ceph-mon[89904]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:52:38.462 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:38 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:52:38.462 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:38 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:38.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:38 smithi186 ceph-mon[67903]: pgmap v310: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 2 op/s 2024-03-22T05:52:38.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:38 smithi186 ceph-mon[67903]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:52:38.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:52:38.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:52:38.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:52:38.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:52:38.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:52:38.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:52:38.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:52:38.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:52:38.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:52:38.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:52:38.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:52:38.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:52:38.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:52:38.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:52:38.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:52:38.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:52:38.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:52:38.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:52:38.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:52:38.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:52:38.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:52:38.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:52:38.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:52:38.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:52:38.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:52:38.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:52:38.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:52:38.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:52:38.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: return future.result(timeout) 2024-03-22T05:52:38.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:52:38.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.__get_result() 2024-03-22T05:52:38.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:52:38.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: raise self._exception 2024-03-22T05:52:38.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:52:38.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:52:38.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:52:38.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:52:38.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:52:38.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:52:38.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:52:38.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:52:38.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:52:38.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:52:38.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:52:38.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:52:38.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:52:38.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:52:38.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:52:38.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:52:38.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:52:38.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:52:38.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:52:38.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:52:38.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:52:38.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:52:38.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:52:38.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:52:38.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:52:38.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:38 smithi186 ceph-mon[67903]: pgmap v311: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 2 op/s 2024-03-22T05:52:38.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:38 smithi186 ceph-mon[67903]: pgmap v312: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 2 op/s 2024-03-22T05:52:38.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:38 smithi186 ceph-mon[67903]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:52:38.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:38 smithi186 ceph-mon[67903]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:52:38.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:38 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:52:38.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:38 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:52:38.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:52:38.711+0000 7f7f0ecc9700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:52:38.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:52:38.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:52:38.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:52:38.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:52:38.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:52:38.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:52:38.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:52:38.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:52:38.953 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:52:38.954 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:52:38.954 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:52:38.954 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:52:38.954 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:52:38.954 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:52:38.954 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:52:38.954 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:52:38.954 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:52:38.954 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:52:38.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:52:38.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:52:38.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:52:38.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:52:38.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:52:38.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:52:38.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:52:38.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:52:38.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:52:38.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:52:38.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return future.result(timeout) 2024-03-22T05:52:38.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:52:38.955 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.__get_result() 2024-03-22T05:52:38.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:52:38.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: raise self._exception 2024-03-22T05:52:38.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:52:38.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: config_blobs=daemon_spec.final_config, 2024-03-22T05:52:38.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:52:38.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:52:38.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:52:38.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:52:38.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:52:38.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:52:38.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:52:38.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:52:38.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:52:38.956 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:52:38.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:52:38.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:52:38.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:52:38.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:52:38.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:52:38.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:52:38.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:52:38.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:52:38.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:52:38.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:52:38.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:52:38.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:52:38.957 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:38 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:52:40.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:40 smithi005 ceph-mon[84705]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:52:40.453 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:52:40.453 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:52:40.453 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:52:40.453 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:52:40.453 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:52:40.453 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:52:40.453 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:52:40.453 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:52:40.453 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:52:40.453 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:52:40.453 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:52:40.454 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:52:40.454 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:52:40.454 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:52:40.454 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:52:40.454 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:52:40.454 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:52:40.454 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:52:40.454 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:52:40.454 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:52:40.454 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:52:40.454 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:52:40.454 INFO:journalctl@ceph.mon.a.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:52:40.454 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:52:40.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:52:40.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:52:40.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:52:40.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:52:40.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: return future.result(timeout) 2024-03-22T05:52:40.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:52:40.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.__get_result() 2024-03-22T05:52:40.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:52:40.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: raise self._exception 2024-03-22T05:52:40.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:52:40.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:52:40.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:52:40.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:52:40.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:52:40.455 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:52:40.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:52:40.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:52:40.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:52:40.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:52:40.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:52:40.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:52:40.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:52:40.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:52:40.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:52:40.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:52:40.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:52:40.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:52:40.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:52:40.456 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:52:40.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:52:40.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:52:40.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:52:40.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:52:40.457 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:52:40.457 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:40 smithi005 ceph-mon[84705]: pgmap v313: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 3 op/s 2024-03-22T05:52:40.457 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:40 smithi005 ceph-mon[89904]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:52:40.457 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:52:40.457 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:52:40.457 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:52:40.457 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:52:40.457 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:52:40.458 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:52:40.458 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:52:40.458 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:52:40.458 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:52:40.458 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:52:40.458 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:52:40.458 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:52:40.458 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:52:40.458 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:52:40.458 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:52:40.458 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:52:40.458 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:52:40.458 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:52:40.458 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:52:40.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:52:40.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:52:40.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:52:40.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:52:40.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:52:40.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:52:40.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:52:40.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:52:40.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:52:40.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: return future.result(timeout) 2024-03-22T05:52:40.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:52:40.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.__get_result() 2024-03-22T05:52:40.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:52:40.459 INFO:journalctl@ceph.mon.c.smithi005.stdout: raise self._exception 2024-03-22T05:52:40.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:52:40.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:52:40.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:52:40.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:52:40.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:52:40.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:52:40.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:52:40.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:52:40.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:52:40.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:52:40.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:52:40.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:52:40.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:52:40.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:52:40.460 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:52:40.461 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:52:40.461 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:52:40.461 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:52:40.461 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:52:40.461 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:52:40.461 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:52:40.461 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:52:40.461 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:52:40.461 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:52:40.461 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:52:40.461 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:40 smithi005 ceph-mon[89904]: pgmap v313: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 3 op/s 2024-03-22T05:52:40.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:40 smithi186 ceph-mon[67903]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:52:40.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:52:40.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:52:40.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:52:40.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:52:40.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:52:40.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:52:40.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:52:40.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:52:40.544 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:52:40.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:52:40.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:52:40.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:52:40.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:52:40.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:52:40.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:52:40.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:52:40.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:52:40.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:52:40.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:52:40.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:52:40.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:52:40.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:52:40.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:52:40.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:52:40.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:52:40.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:52:40.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:52:40.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:52:40.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: return future.result(timeout) 2024-03-22T05:52:40.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:52:40.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.__get_result() 2024-03-22T05:52:40.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:52:40.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: raise self._exception 2024-03-22T05:52:40.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:52:40.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:52:40.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:52:40.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:52:40.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:52:40.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:52:40.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:52:40.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:52:40.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:52:40.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:52:40.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:52:40.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:52:40.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:52:40.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:52:40.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:52:40.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:52:40.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:52:40.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:52:40.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:52:40.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:52:40.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:52:40.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:52:40.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:52:40.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:52:40.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:52:40.548 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:40 smithi186 ceph-mon[67903]: pgmap v313: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 3 op/s 2024-03-22T05:52:42.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:42 smithi005 ceph-mon[84705]: pgmap v314: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 4.5 KiB/s rd, 4 op/s 2024-03-22T05:52:42.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:42 smithi005 ceph-mon[89904]: pgmap v314: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 4.5 KiB/s rd, 4 op/s 2024-03-22T05:52:42.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:42 smithi186 ceph-mon[67903]: pgmap v314: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 4.5 KiB/s rd, 4 op/s 2024-03-22T05:52:42.614 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph orch ps' 2024-03-22T05:52:43.193 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:43 smithi005 ceph-mon[84705]: from='client.54762 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:52:43.194 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:43 smithi005 ceph-mon[89904]: from='client.54762 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:52:43.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:43 smithi186 ceph-mon[67903]: from='client.54762 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:52:44.044 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:52:43 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=cleanup t=2024-03-22T05:52:43.649068437Z level=info msg="Completed cleanup jobs" duration=1.130236ms 2024-03-22T05:52:44.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:44 smithi005 ceph-mon[84705]: pgmap v315: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 4 op/s 2024-03-22T05:52:44.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:44 smithi005 ceph-mon[89904]: pgmap v315: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 4 op/s 2024-03-22T05:52:44.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:44 smithi186 ceph-mon[67903]: pgmap v315: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 4 op/s 2024-03-22T05:52:44.720 INFO:teuthology.orchestra.run.smithi005.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T05:52:44.720 INFO:teuthology.orchestra.run.smithi005.stdout:alertmanager.a smithi005 *:9093,9094 running (10m) 16s ago 32m 27.2M - 0.25.0 c8568f914cd2 ee99dfc435e8 2024-03-22T05:52:44.720 INFO:teuthology.orchestra.run.smithi005.stdout:grafana.a smithi186 *:3000 running (10m) 17s ago 31m 83.5M - 9.4.12 2bacad6d85d8 16fd57dbbd71 2024-03-22T05:52:44.720 INFO:teuthology.orchestra.run.smithi005.stdout:iscsi.foo.smithi005.mcdwbp smithi005 unknown 16s ago 31m 64.7M - 3.5 e1d6a67b021e 5215e03bf81b 2024-03-22T05:52:44.720 INFO:teuthology.orchestra.run.smithi005.stdout:mgr.x smithi186 *:8443,9283,8765 running (11m) 17s ago 36m 454M - 19.0.0-2299-gea01217f aa461e7c5e7f c556421a73f3 2024-03-22T05:52:44.720 INFO:teuthology.orchestra.run.smithi005.stdout:mgr.y smithi005 *:8443,9283,8765 running (21m) 16s ago 38m 517M - 19.0.0-2299-gea01217f aa461e7c5e7f 62a813eebc0d 2024-03-22T05:52:44.720 INFO:teuthology.orchestra.run.smithi005.stdout:mon.a smithi005 running (8m) 16s ago 38m 63.0M 2048M 19.0.0-2299-gea01217f aa461e7c5e7f 111a2aa24dd9 2024-03-22T05:52:44.720 INFO:teuthology.orchestra.run.smithi005.stdout:mon.b smithi186 running (8m) 17s ago 36m 55.8M 2048M 19.0.0-2299-gea01217f aa461e7c5e7f c1798feac15b 2024-03-22T05:52:44.720 INFO:teuthology.orchestra.run.smithi005.stdout:mon.c smithi005 running (7m) 16s ago 37m 48.4M 2048M 19.0.0-2299-gea01217f aa461e7c5e7f 708c72ff74f5 2024-03-22T05:52:44.720 INFO:teuthology.orchestra.run.smithi005.stdout:node-exporter.a smithi005 *:9100 running (11m) 16s ago 32m 16.5M - 1.5.0 0da6a335fe13 f635cd72b9d3 2024-03-22T05:52:44.720 INFO:teuthology.orchestra.run.smithi005.stdout:node-exporter.b smithi186 *:9100 running (10m) 17s ago 32m 14.7M - 1.5.0 0da6a335fe13 d7e0b3d20191 2024-03-22T05:52:44.721 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0 smithi005 running (5m) 16s ago 36m 54.4M 2993M 19.0.0-2299-gea01217f aa461e7c5e7f 31b6cdee5f5b 2024-03-22T05:52:44.721 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1 smithi005 running (5m) 16s ago 35m 54.9M 2993M 19.0.0-2299-gea01217f aa461e7c5e7f 7d21bc9fd3c2 2024-03-22T05:52:44.721 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2 smithi005 running (4m) 16s ago 35m 49.6M 2993M 19.0.0-2299-gea01217f aa461e7c5e7f 628204cb7c98 2024-03-22T05:52:44.721 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3 smithi005 running (3m) 16s ago 35m 64.0M 2993M 19.0.0-2299-gea01217f aa461e7c5e7f d31a01a0230c 2024-03-22T05:52:44.721 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4 smithi186 running (2m) 17s ago 34m 63.4M 3248M 19.0.0-2299-gea01217f aa461e7c5e7f 8f7619b726f3 2024-03-22T05:52:44.721 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5 smithi186 running (2m) 17s ago 34m 53.8M 3248M 19.0.0-2299-gea01217f aa461e7c5e7f b80f7eceb2d7 2024-03-22T05:52:44.721 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6 smithi186 running (96s) 17s ago 34m 55.5M 3248M 19.0.0-2299-gea01217f aa461e7c5e7f db7c81962c65 2024-03-22T05:52:44.721 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7 smithi186 running (74s) 17s ago 33m 50.0M 3248M 19.0.0-2299-gea01217f aa461e7c5e7f fcb46ba843ae 2024-03-22T05:52:44.721 INFO:teuthology.orchestra.run.smithi005.stdout:prometheus.a smithi186 *:9095 running (10m) 17s ago 32m 72.0M - 2.43.0 a07b618ecd1d a5a043588d6d 2024-03-22T05:52:44.721 INFO:teuthology.orchestra.run.smithi005.stdout:rgw.foo.smithi005.exdhqu smithi005 *:8000 running (27s) 16s ago 31m 76.1M - 19.0.0-2299-gea01217f aa461e7c5e7f aa3dec0ece5c 2024-03-22T05:52:44.721 INFO:teuthology.orchestra.run.smithi005.stdout:rgw.foo.smithi186.mcgodk smithi186 *:8000 running (21s) 17s ago 31m 75.9M - 19.0.0-2299-gea01217f aa461e7c5e7f c7967b92514a 2024-03-22T05:52:45.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:45 smithi005 ceph-mon[84705]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:52:45.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:45 smithi005 ceph-mon[89904]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:52:45.481 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph versions | jq -e '"'"'.rgw | length == 1'"'"'' 2024-03-22T05:52:45.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:45 smithi186 ceph-mon[67903]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:52:46.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:46 smithi005 ceph-mon[84705]: from='client.54768 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:52:46.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:46 smithi005 ceph-mon[84705]: pgmap v316: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-03-22T05:52:46.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:46 smithi005 ceph-mon[89904]: from='client.54768 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:52:46.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:46 smithi005 ceph-mon[89904]: pgmap v316: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-03-22T05:52:46.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:46 smithi186 ceph-mon[67903]: from='client.54768 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:52:46.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:46 smithi186 ceph-mon[67903]: pgmap v316: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-03-22T05:52:47.121 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:46 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:52:46] "GET /metrics HTTP/1.1" 200 38393 "" "Prometheus/2.43.0" 2024-03-22T05:52:47.907 INFO:teuthology.orchestra.run.smithi005.stdout:true 2024-03-22T05:52:48.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:48 smithi005 ceph-mon[84705]: pgmap v317: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-03-22T05:52:48.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:48 smithi005 ceph-mon[84705]: from='client.? 172.21.15.5:0/1366512392' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:52:48.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:48 smithi005 ceph-mon[89904]: pgmap v317: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-03-22T05:52:48.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:48 smithi005 ceph-mon[89904]: from='client.? 172.21.15.5:0/1366512392' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:52:48.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:48 smithi186 ceph-mon[67903]: pgmap v317: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-03-22T05:52:48.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:48 smithi186 ceph-mon[67903]: from='client.? 172.21.15.5:0/1366512392' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:52:48.614 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph versions | jq -e '"'"'.rgw | keys'"'"' | grep $sha1' 2024-03-22T05:52:50.318 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:50 smithi005 ceph-mon[84705]: pgmap v318: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 2 op/s 2024-03-22T05:52:50.318 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:50 smithi005 ceph-mon[89904]: pgmap v318: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 2 op/s 2024-03-22T05:52:50.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:50 smithi186 ceph-mon[67903]: pgmap v318: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 2 op/s 2024-03-22T05:52:50.863 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev)" 2024-03-22T05:52:51.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:51 smithi005 ceph-mon[89904]: from='client.? 172.21.15.5:0/2898806097' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:52:51.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:51 smithi005 ceph-mon[84705]: from='client.? 172.21.15.5:0/2898806097' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:52:51.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:51 smithi186 ceph-mon[67903]: from='client.? 172.21.15.5:0/2898806097' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:52:51.593 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph orch upgrade status' 2024-03-22T05:52:52.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:52 smithi005 ceph-mon[84705]: pgmap v319: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-03-22T05:52:52.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:52 smithi005 ceph-mon[89904]: pgmap v319: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-03-22T05:52:52.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:52 smithi186 ceph-mon[67903]: pgmap v319: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-03-22T05:52:53.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:53 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:52:53.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:53 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:52:53.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:53 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:52:53.619 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-03-22T05:52:53.619 INFO:teuthology.orchestra.run.smithi005.stdout: "target_image": null, 2024-03-22T05:52:53.619 INFO:teuthology.orchestra.run.smithi005.stdout: "in_progress": false, 2024-03-22T05:52:53.619 INFO:teuthology.orchestra.run.smithi005.stdout: "which": "", 2024-03-22T05:52:53.619 INFO:teuthology.orchestra.run.smithi005.stdout: "services_complete": [], 2024-03-22T05:52:53.619 INFO:teuthology.orchestra.run.smithi005.stdout: "progress": null, 2024-03-22T05:52:53.619 INFO:teuthology.orchestra.run.smithi005.stdout: "message": "", 2024-03-22T05:52:53.619 INFO:teuthology.orchestra.run.smithi005.stdout: "is_paused": false 2024-03-22T05:52:53.619 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-03-22T05:52:54.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:54 smithi005 ceph-mon[84705]: pgmap v320: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:52:54.203 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:54 smithi005 ceph-mon[89904]: pgmap v320: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:52:54.398 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph health detail' 2024-03-22T05:52:54.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:54 smithi186 ceph-mon[67903]: pgmap v320: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:52:55.348 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:55 smithi005 ceph-mon[84705]: from='client.54786 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:52:55.348 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:55 smithi005 ceph-mon[84705]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:52:55.348 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:55 smithi005 ceph-mon[89904]: from='client.54786 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:52:55.348 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:55 smithi005 ceph-mon[89904]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:52:55.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:55 smithi186 ceph-mon[67903]: from='client.54786 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:52:55.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:55 smithi186 ceph-mon[67903]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:52:56.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:56 smithi005 ceph-mon[84705]: pgmap v321: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:52:56.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:56 smithi005 ceph-mon[89904]: pgmap v321: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:52:56.527 INFO:teuthology.orchestra.run.smithi005.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-22T05:52:56.527 INFO:teuthology.orchestra.run.smithi005.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-22T05:52:56.527 INFO:teuthology.orchestra.run.smithi005.stdout: daemon iscsi.foo.smithi005.mcdwbp on smithi005 is in unknown state 2024-03-22T05:52:56.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:56 smithi186 ceph-mon[67903]: pgmap v321: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:52:57.180 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:52:56 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:52:56] "GET /metrics HTTP/1.1" 200 38398 "" "Prometheus/2.43.0" 2024-03-22T05:52:57.255 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-03-22T05:52:57.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:57 smithi005 ceph-mon[84705]: from='client.? 172.21.15.5:0/664664533' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T05:52:57.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:57 smithi005 ceph-mon[89904]: from='client.? 172.21.15.5:0/664664533' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T05:52:57.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:57 smithi186 ceph-mon[67903]: from='client.? 172.21.15.5:0/664664533' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T05:52:58.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:52:58 smithi005 ceph-mon[84705]: pgmap v322: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:52:58.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:52:58 smithi005 ceph-mon[89904]: pgmap v322: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:52:58.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:52:58 smithi186 ceph-mon[67903]: pgmap v322: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:52:59.481 INFO:teuthology.orchestra.run.smithi005.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37 2024-03-22T05:53:00.199 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-22T05:53:00.208 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi005.front.sepia.ceph.com 2024-03-22T05:53:00.209 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-03-22T05:53:00.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:00 smithi005 ceph-mon[84705]: pgmap v323: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:53:00.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:00 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:53:00.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:00 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:53:00.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:00 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:53:00.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:00 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:53:00.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:00 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:53:00.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:00 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:53:00.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:00 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:53:00.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:00 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:53:00.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:00 smithi005 ceph-mon[89904]: pgmap v323: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:53:00.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:00 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:53:00.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:00 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:53:00.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:00 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:53:00.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:00 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:53:00.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:00 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:53:00.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:00 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:53:00.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:00 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:53:00.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:00 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:53:00.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:00 smithi186 ceph-mon[67903]: pgmap v323: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:53:00.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:00 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:53:00.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:00 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:53:00.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:00 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:53:00.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:00 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:53:00.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:00 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:53:00.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:00 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:53:00.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:00 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:53:00.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:00 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:53:01.385 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:01 smithi005 ceph-mon[84705]: from='client.64723 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:53:01.385 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:01 smithi005 ceph-mon[84705]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37 2024-03-22T05:53:01.385 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:01 smithi005 ceph-mon[84705]: pgmap v324: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 955 B/s rd, 0 op/s 2024-03-22T05:53:01.385 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:01 smithi005 ceph-mon[84705]: pgmap v325: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:53:01.385 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:01 smithi005 ceph-mon[84705]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:53:01.385 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:01 smithi005 ceph-mon[84705]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:53:01.385 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:01 smithi005 ceph-mon[89904]: from='client.64723 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:53:01.386 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:01 smithi005 ceph-mon[89904]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37 2024-03-22T05:53:01.386 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:01 smithi005 ceph-mon[89904]: pgmap v324: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 955 B/s rd, 0 op/s 2024-03-22T05:53:01.386 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:01 smithi005 ceph-mon[89904]: pgmap v325: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:53:01.386 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:01 smithi005 ceph-mon[89904]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:53:01.386 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:01 smithi005 ceph-mon[89904]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:53:01.513 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:01 smithi186 ceph-mon[67903]: from='client.64723 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:53:01.514 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:01 smithi186 ceph-mon[67903]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37 2024-03-22T05:53:01.514 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:01 smithi186 ceph-mon[67903]: pgmap v324: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 955 B/s rd, 0 op/s 2024-03-22T05:53:01.514 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:01 smithi186 ceph-mon[67903]: pgmap v325: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:53:01.514 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:01 smithi186 ceph-mon[67903]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:53:01.514 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:01 smithi186 ceph-mon[67903]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:53:02.203 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:53:01 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:53:01.800+0000 7f7f0ecc9700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:53:02.203 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:53:01 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:53:02.203 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:53:01 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:53:02.203 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:53:01 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:53:02.204 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:53:01 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:53:02.204 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:53:01 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:53:02.204 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:53:01 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:53:02.204 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:53:01 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:53:02.204 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:53:01 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:53:02.204 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:53:01 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:53:02.204 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:53:01 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:53:02.204 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:53:01 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:53:02.204 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:53:01 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:53:02.204 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:53:01 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:53:02.204 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:53:01 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:53:02.204 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:53:01 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:53:02.204 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:53:01 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:53:02.205 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:53:01 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:53:02.205 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:53:01 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:53:02.205 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:53:01 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:53:02.205 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:53:01 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:53:02.205 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:53:01 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:53:02.205 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:53:01 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:53:02.206 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:53:01 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:53:02.206 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:53:01 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:53:02.206 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:53:01 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:53:02.206 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:53:01 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:53:02.206 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:53:01 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:53:02.206 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:53:01 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:53:02.206 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:53:01 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return future.result(timeout) 2024-03-22T05:53:02.206 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:53:01 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:53:02.206 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:53:01 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return self.__get_result() 2024-03-22T05:53:02.206 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:53:01 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:53:02.206 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:53:01 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: raise self._exception 2024-03-22T05:53:02.206 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:53:01 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:53:02.207 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:53:01 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: config_blobs=daemon_spec.final_config, 2024-03-22T05:53:02.207 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:53:01 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:53:02.207 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:53:01 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:53:02.207 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:53:01 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:53:02.207 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:53:01 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Creating ceph-iscsi config... 2024-03-22T05:53:02.207 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:53:01 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:53:02.207 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:53:01 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:53:02.207 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:53:01 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:53:02.207 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:53:01 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:53:02.207 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:53:01 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:53:02.207 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:53:01 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: Traceback (most recent call last): 2024-03-22T05:53:02.207 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:53:01 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:53:02.207 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:53:01 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: return _run_code(code, main_globals, None, 2024-03-22T05:53:02.207 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:53:01 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:53:02.207 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:53:01 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: exec(code, run_globals) 2024-03-22T05:53:02.208 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:53:01 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:53:02.208 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:53:01 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:53:02.208 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:53:01 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:53:02.208 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:53:01 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:53:02.208 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:53:01 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:53:02.208 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:53:01 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:53:02.208 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:53:01 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:53:02.208 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:53:01 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:53:02.208 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:53:01 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:53:03.325 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:03 smithi005 ceph-mon[89904]: pgmap v326: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 587 B/s rd, 0 op/s 2024-03-22T05:53:03.325 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:03 smithi005 ceph-mon[89904]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:53:03.325 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:53:03.325 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:53:03.325 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:53:03.325 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:53:03.325 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:53:03.325 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:53:03.325 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:53:03.326 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:53:03.326 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:53:03.326 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:53:03.326 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:53:03.326 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:53:03.326 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:53:03.326 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:53:03.326 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:53:03.326 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:53:03.326 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:53:03.326 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:53:03.326 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:53:03.326 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:53:03.326 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:53:03.326 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:53:03.327 INFO:journalctl@ceph.mon.c.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:53:03.327 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:53:03.327 INFO:journalctl@ceph.mon.c.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:53:03.327 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:53:03.327 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:53:03.327 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:53:03.327 INFO:journalctl@ceph.mon.c.smithi005.stdout: return future.result(timeout) 2024-03-22T05:53:03.327 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:53:03.327 INFO:journalctl@ceph.mon.c.smithi005.stdout: return self.__get_result() 2024-03-22T05:53:03.327 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:53:03.327 INFO:journalctl@ceph.mon.c.smithi005.stdout: raise self._exception 2024-03-22T05:53:03.327 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:53:03.327 INFO:journalctl@ceph.mon.c.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:53:03.327 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:53:03.327 INFO:journalctl@ceph.mon.c.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:53:03.327 INFO:journalctl@ceph.mon.c.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:53:03.328 INFO:journalctl@ceph.mon.c.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:53:03.328 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:53:03.328 INFO:journalctl@ceph.mon.c.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:53:03.328 INFO:journalctl@ceph.mon.c.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:53:03.328 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:53:03.328 INFO:journalctl@ceph.mon.c.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:53:03.328 INFO:journalctl@ceph.mon.c.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:53:03.328 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:53:03.328 INFO:journalctl@ceph.mon.c.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:53:03.328 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:53:03.328 INFO:journalctl@ceph.mon.c.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:53:03.328 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:53:03.328 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:53:03.328 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:53:03.328 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:53:03.329 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:03 smithi005 ceph-mon[84705]: pgmap v326: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 587 B/s rd, 0 op/s 2024-03-22T05:53:03.329 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:03 smithi005 ceph-mon[84705]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:53:03.329 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:53:03.329 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:53:03.329 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:53:03.329 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:53:03.329 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:53:03.329 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:53:03.329 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:53:03.329 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:53:03.330 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:53:03.330 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:53:03.330 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:53:03.330 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:53:03.330 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:53:03.330 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:53:03.330 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:53:03.330 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:53:03.330 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:53:03.330 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:53:03.330 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:53:03.330 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:53:03.330 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:53:03.330 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:53:03.331 INFO:journalctl@ceph.mon.a.smithi005.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:53:03.331 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:53:03.331 INFO:journalctl@ceph.mon.a.smithi005.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:53:03.331 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:53:03.331 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:53:03.331 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:53:03.331 INFO:journalctl@ceph.mon.a.smithi005.stdout: return future.result(timeout) 2024-03-22T05:53:03.331 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:53:03.331 INFO:journalctl@ceph.mon.a.smithi005.stdout: return self.__get_result() 2024-03-22T05:53:03.331 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:53:03.331 INFO:journalctl@ceph.mon.a.smithi005.stdout: raise self._exception 2024-03-22T05:53:03.331 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:53:03.331 INFO:journalctl@ceph.mon.a.smithi005.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:53:03.331 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:53:03.331 INFO:journalctl@ceph.mon.a.smithi005.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:53:03.331 INFO:journalctl@ceph.mon.a.smithi005.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:53:03.332 INFO:journalctl@ceph.mon.a.smithi005.stdout: Creating ceph-iscsi config... 2024-03-22T05:53:03.332 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:53:03.332 INFO:journalctl@ceph.mon.a.smithi005.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:53:03.332 INFO:journalctl@ceph.mon.a.smithi005.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:53:03.332 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:53:03.332 INFO:journalctl@ceph.mon.a.smithi005.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:53:03.332 INFO:journalctl@ceph.mon.a.smithi005.stdout: Traceback (most recent call last): 2024-03-22T05:53:03.332 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:53:03.332 INFO:journalctl@ceph.mon.a.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:53:03.332 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:53:03.332 INFO:journalctl@ceph.mon.a.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:53:03.332 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:53:03.332 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:53:03.332 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:53:03.332 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:53:03.333 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:53:03.333 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:53:03.333 INFO:journalctl@ceph.mon.c.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:53:03.333 INFO:journalctl@ceph.mon.c.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:53:03.333 INFO:journalctl@ceph.mon.c.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:53:03.333 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:03 smithi005 ceph-mon[89904]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37 2024-03-22T05:53:03.333 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:53:03.333 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:53:03.333 INFO:journalctl@ceph.mon.a.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:53:03.334 INFO:journalctl@ceph.mon.a.smithi005.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:53:03.334 INFO:journalctl@ceph.mon.a.smithi005.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:53:03.334 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:03 smithi005 ceph-mon[84705]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37 2024-03-22T05:53:03.446 INFO:teuthology.orchestra.run.smithi005.stdout:true 2024-03-22T05:53:03.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:03 smithi186 ceph-mon[67903]: pgmap v326: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 587 B/s rd, 0 op/s 2024-03-22T05:53:03.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:03 smithi186 ceph-mon[67903]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:53:03.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:53:03.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:53:03.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:53:03.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:53:03.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:53:03.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:53:03.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:53:03.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:53:03.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:53:03.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:53:03.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:53:03.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:53:03.545 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:53:03.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:53:03.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:53:03.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:53:03.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:53:03.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:53:03.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:53:03.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:53:03.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:53:03.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-22T05:53:03.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-22T05:53:03.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2424, in _daemon_action 2024-03-22T05:53:03.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-22T05:53:03.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-22T05:53:03.546 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-22T05:53:03.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-22T05:53:03.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: return future.result(timeout) 2024-03-22T05:53:03.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-22T05:53:03.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: return self.__get_result() 2024-03-22T05:53:03.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-22T05:53:03.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: raise self._exception 2024-03-22T05:53:03.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-22T05:53:03.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: config_blobs=daemon_spec.final_config, 2024-03-22T05:53:03.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-22T05:53:03.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-22T05:53:03.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:53:03.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: Creating ceph-iscsi config... 2024-03-22T05:53:03.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:53:03.547 INFO:journalctl@ceph.mon.b.smithi186.stdout: Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:53:03.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: Non-zero exit code 1 from systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:53:03.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:53:03.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:53:03.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: Traceback (most recent call last): 2024-03-22T05:53:03.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:53:03.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:53:03.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:53:03.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: exec(code, run_globals) 2024-03-22T05:53:03.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:53:03.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:53:03.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:53:03.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:53:03.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:53:03.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1123, in deploy_daemon 2024-03-22T05:53:03.548 INFO:journalctl@ceph.mon.b.smithi186.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:53:03.549 INFO:journalctl@ceph.mon.b.smithi186.stdout: RuntimeError: Failed command: systemctl restart ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:53:03.549 INFO:journalctl@ceph.mon.b.smithi186.stdout: See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:53:03.549 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:03 smithi186 ceph-mon[67903]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:ea01217f8a79678e64ef6a5d68a043a4ef21ca37 2024-03-22T05:53:04.157 INFO:teuthology.orchestra.run.smithi005.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T05:53:04.157 INFO:teuthology.orchestra.run.smithi005.stdout:alertmanager.a smithi005 *:9093,9094 running (10m) 36s ago 32m 27.2M - 0.25.0 c8568f914cd2 ee99dfc435e8 2024-03-22T05:53:04.157 INFO:teuthology.orchestra.run.smithi005.stdout:grafana.a smithi186 *:3000 running (10m) 36s ago 32m 83.5M - 9.4.12 2bacad6d85d8 16fd57dbbd71 2024-03-22T05:53:04.157 INFO:teuthology.orchestra.run.smithi005.stdout:iscsi.foo.smithi005.mcdwbp smithi005 unknown 36s ago 31m 64.7M - 3.5 e1d6a67b021e 5215e03bf81b 2024-03-22T05:53:04.157 INFO:teuthology.orchestra.run.smithi005.stdout:mgr.x smithi186 *:8443,9283,8765 running (11m) 36s ago 36m 454M - 19.0.0-2299-gea01217f aa461e7c5e7f c556421a73f3 2024-03-22T05:53:04.157 INFO:teuthology.orchestra.run.smithi005.stdout:mgr.y smithi005 *:8443,9283,8765 running (21m) 36s ago 38m 517M - 19.0.0-2299-gea01217f aa461e7c5e7f 62a813eebc0d 2024-03-22T05:53:04.157 INFO:teuthology.orchestra.run.smithi005.stdout:mon.a smithi005 running (8m) 36s ago 38m 63.0M 2048M 19.0.0-2299-gea01217f aa461e7c5e7f 111a2aa24dd9 2024-03-22T05:53:04.157 INFO:teuthology.orchestra.run.smithi005.stdout:mon.b smithi186 running (9m) 36s ago 37m 55.8M 2048M 19.0.0-2299-gea01217f aa461e7c5e7f c1798feac15b 2024-03-22T05:53:04.157 INFO:teuthology.orchestra.run.smithi005.stdout:mon.c smithi005 running (7m) 36s ago 37m 48.4M 2048M 19.0.0-2299-gea01217f aa461e7c5e7f 708c72ff74f5 2024-03-22T05:53:04.157 INFO:teuthology.orchestra.run.smithi005.stdout:node-exporter.a smithi005 *:9100 running (11m) 36s ago 33m 16.5M - 1.5.0 0da6a335fe13 f635cd72b9d3 2024-03-22T05:53:04.157 INFO:teuthology.orchestra.run.smithi005.stdout:node-exporter.b smithi186 *:9100 running (11m) 36s ago 33m 14.7M - 1.5.0 0da6a335fe13 d7e0b3d20191 2024-03-22T05:53:04.157 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0 smithi005 running (6m) 36s ago 36m 54.4M 2993M 19.0.0-2299-gea01217f aa461e7c5e7f 31b6cdee5f5b 2024-03-22T05:53:04.157 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1 smithi005 running (5m) 36s ago 36m 54.9M 2993M 19.0.0-2299-gea01217f aa461e7c5e7f 7d21bc9fd3c2 2024-03-22T05:53:04.157 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2 smithi005 running (4m) 36s ago 35m 49.6M 2993M 19.0.0-2299-gea01217f aa461e7c5e7f 628204cb7c98 2024-03-22T05:53:04.158 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3 smithi005 running (3m) 36s ago 35m 64.0M 2993M 19.0.0-2299-gea01217f aa461e7c5e7f d31a01a0230c 2024-03-22T05:53:04.158 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4 smithi186 running (3m) 36s ago 35m 63.4M 3248M 19.0.0-2299-gea01217f aa461e7c5e7f 8f7619b726f3 2024-03-22T05:53:04.158 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5 smithi186 running (2m) 36s ago 34m 53.8M 3248M 19.0.0-2299-gea01217f aa461e7c5e7f b80f7eceb2d7 2024-03-22T05:53:04.158 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6 smithi186 running (115s) 36s ago 34m 55.5M 3248M 19.0.0-2299-gea01217f aa461e7c5e7f db7c81962c65 2024-03-22T05:53:04.158 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7 smithi186 running (94s) 36s ago 33m 50.0M 3248M 19.0.0-2299-gea01217f aa461e7c5e7f fcb46ba843ae 2024-03-22T05:53:04.158 INFO:teuthology.orchestra.run.smithi005.stdout:prometheus.a smithi186 *:9095 running (10m) 36s ago 33m 72.0M - 2.43.0 a07b618ecd1d a5a043588d6d 2024-03-22T05:53:04.158 INFO:teuthology.orchestra.run.smithi005.stdout:rgw.foo.smithi005.exdhqu smithi005 *:8000 running (46s) 36s ago 31m 76.1M - 19.0.0-2299-gea01217f aa461e7c5e7f aa3dec0ece5c 2024-03-22T05:53:04.158 INFO:teuthology.orchestra.run.smithi005.stdout:rgw.foo.smithi186.mcgodk smithi186 *:8000 running (40s) 36s ago 31m 75.9M - 19.0.0-2299-gea01217f aa461e7c5e7f c7967b92514a 2024-03-22T05:53:04.575 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-03-22T05:53:04.576 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": { 2024-03-22T05:53:04.576 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev)": 3 2024-03-22T05:53:04.576 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:53:04.576 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr": { 2024-03-22T05:53:04.576 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev)": 2 2024-03-22T05:53:04.576 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:53:04.576 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": { 2024-03-22T05:53:04.576 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev)": 8 2024-03-22T05:53:04.576 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:53:04.576 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw": { 2024-03-22T05:53:04.576 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev)": 2 2024-03-22T05:53:04.576 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:53:04.576 INFO:teuthology.orchestra.run.smithi005.stdout: "overall": { 2024-03-22T05:53:04.576 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev)": 15 2024-03-22T05:53:04.577 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-03-22T05:53:04.577 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-03-22T05:53:04.934 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-03-22T05:53:04.934 INFO:teuthology.orchestra.run.smithi005.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9", 2024-03-22T05:53:04.934 INFO:teuthology.orchestra.run.smithi005.stdout: "in_progress": true, 2024-03-22T05:53:04.934 INFO:teuthology.orchestra.run.smithi005.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-22T05:53:04.934 INFO:teuthology.orchestra.run.smithi005.stdout: "services_complete": [ 2024-03-22T05:53:04.935 INFO:teuthology.orchestra.run.smithi005.stdout: "mon", 2024-03-22T05:53:04.935 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr", 2024-03-22T05:53:04.935 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw", 2024-03-22T05:53:04.935 INFO:teuthology.orchestra.run.smithi005.stdout: "osd" 2024-03-22T05:53:04.935 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-03-22T05:53:04.935 INFO:teuthology.orchestra.run.smithi005.stdout: "progress": "15/21 daemons upgraded", 2024-03-22T05:53:04.935 INFO:teuthology.orchestra.run.smithi005.stdout: "message": "Currently upgrading iscsi daemons", 2024-03-22T05:53:04.935 INFO:teuthology.orchestra.run.smithi005.stdout: "is_paused": false 2024-03-22T05:53:04.935 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-03-22T05:53:05.367 INFO:teuthology.orchestra.run.smithi005.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-22T05:53:05.367 INFO:teuthology.orchestra.run.smithi005.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-22T05:53:05.367 INFO:teuthology.orchestra.run.smithi005.stdout: daemon iscsi.foo.smithi005.mcdwbp on smithi005 is in unknown state 2024-03-22T05:53:05.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:05 smithi005 ceph-mon[89904]: from='client.64729 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:53:05.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:05 smithi005 ceph-mon[89904]: pgmap v327: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:53:05.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:05 smithi005 ceph-mon[89904]: from='client.64735 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:53:05.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:05 smithi005 ceph-mon[89904]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:53:05.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:05 smithi005 ceph-mon[89904]: from='client.64741 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:53:05.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:05 smithi005 ceph-mon[89904]: from='client.? 172.21.15.5:0/4143871158' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:53:05.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:05 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:53:05.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:05 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:53:05.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:05 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:53:05.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:05 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:53:05.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:05 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:53:05.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:05 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:53:05.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:05 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:53:05.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:05 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:53:05.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:05 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:53:05.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:05 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:53:05.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:05 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:53:05.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:05 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:53:05.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:05 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:53:05.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:05 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:53:05.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:05 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:53:05.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:05 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:53:05.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:05 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:53:05.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:05 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:53:05.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:05 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:53:05.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:05 smithi005 ceph-mon[84705]: from='client.64729 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:53:05.455 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:05 smithi005 ceph-mon[84705]: pgmap v327: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:53:05.455 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:05 smithi005 ceph-mon[84705]: from='client.64735 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:53:05.455 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:05 smithi005 ceph-mon[84705]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:53:05.455 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:05 smithi005 ceph-mon[84705]: from='client.64741 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:53:05.455 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:05 smithi005 ceph-mon[84705]: from='client.? 172.21.15.5:0/4143871158' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:53:05.455 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:05 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:53:05.455 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:05 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:53:05.455 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:05 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:53:05.455 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:05 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:53:05.455 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:05 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:53:05.455 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:05 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:53:05.455 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:05 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:53:05.455 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:05 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:53:05.456 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:05 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:53:05.456 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:05 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:53:05.456 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:05 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:53:05.456 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:05 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:53:05.456 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:05 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:53:05.456 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:05 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:53:05.456 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:05 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:53:05.456 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:05 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:53:05.456 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:05 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:53:05.456 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:05 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:53:05.456 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:05 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:53:05.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:05 smithi186 ceph-mon[67903]: from='client.64729 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:53:05.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:05 smithi186 ceph-mon[67903]: pgmap v327: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:53:05.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:05 smithi186 ceph-mon[67903]: from='client.64735 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:53:05.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:05 smithi186 ceph-mon[67903]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:53:05.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:05 smithi186 ceph-mon[67903]: from='client.64741 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:53:05.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:05 smithi186 ceph-mon[67903]: from='client.? 172.21.15.5:0/4143871158' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:53:05.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:05 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:53:05.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:05 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:53:05.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:05 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:53:05.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:05 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:53:05.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:05 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:53:05.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:05 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:53:05.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:05 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:53:05.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:05 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:53:05.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:05 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:53:05.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:05 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:53:05.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:05 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:53:05.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:05 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:53:05.546 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:05 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:53:05.546 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:05 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:53:05.546 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:05 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:53:05.546 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:05 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:53:05.546 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:05 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:53:05.546 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:05 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:53:05.546 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:05 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:53:06.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:06 smithi005 ceph-mon[89904]: Upgrade: Target is version 19.0.0-2299-gea01217f (squid) 2024-03-22T05:53:06.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:06 smithi005 ceph-mon[89904]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, digests ['quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9'] 2024-03-22T05:53:06.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:06 smithi005 ceph-mon[89904]: Upgrade: Setting container_image for all mgr 2024-03-22T05:53:06.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:06 smithi005 ceph-mon[89904]: Upgrade: Setting container_image for all mon 2024-03-22T05:53:06.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:06 smithi005 ceph-mon[89904]: Upgrade: Setting container_image for all crash 2024-03-22T05:53:06.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:06 smithi005 ceph-mon[89904]: Upgrade: Setting container_image for all osd 2024-03-22T05:53:06.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:06 smithi005 ceph-mon[89904]: Upgrade: Setting container_image for all mds 2024-03-22T05:53:06.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:06 smithi005 ceph-mon[89904]: Upgrade: Setting container_image for all rgw 2024-03-22T05:53:06.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:06 smithi005 ceph-mon[89904]: Upgrade: Setting container_image for all rbd-mirror 2024-03-22T05:53:06.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:06 smithi005 ceph-mon[89904]: Upgrade: Setting container_image for all cephfs-mirror 2024-03-22T05:53:06.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:06 smithi005 ceph-mon[89904]: Upgrade: Setting container_image for all ceph-exporter 2024-03-22T05:53:06.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:06 smithi005 ceph-mon[89904]: from='client.64753 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:53:06.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:06 smithi005 ceph-mon[89904]: from='client.? 172.21.15.5:0/738498728' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T05:53:06.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:06 smithi005 ceph-mon[84705]: Upgrade: Target is version 19.0.0-2299-gea01217f (squid) 2024-03-22T05:53:06.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:06 smithi005 ceph-mon[84705]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, digests ['quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9'] 2024-03-22T05:53:06.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:06 smithi005 ceph-mon[84705]: Upgrade: Setting container_image for all mgr 2024-03-22T05:53:06.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:06 smithi005 ceph-mon[84705]: Upgrade: Setting container_image for all mon 2024-03-22T05:53:06.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:06 smithi005 ceph-mon[84705]: Upgrade: Setting container_image for all crash 2024-03-22T05:53:06.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:06 smithi005 ceph-mon[84705]: Upgrade: Setting container_image for all osd 2024-03-22T05:53:06.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:06 smithi005 ceph-mon[84705]: Upgrade: Setting container_image for all mds 2024-03-22T05:53:06.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:06 smithi005 ceph-mon[84705]: Upgrade: Setting container_image for all rgw 2024-03-22T05:53:06.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:06 smithi005 ceph-mon[84705]: Upgrade: Setting container_image for all rbd-mirror 2024-03-22T05:53:06.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:06 smithi005 ceph-mon[84705]: Upgrade: Setting container_image for all cephfs-mirror 2024-03-22T05:53:06.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:06 smithi005 ceph-mon[84705]: Upgrade: Setting container_image for all ceph-exporter 2024-03-22T05:53:06.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:06 smithi005 ceph-mon[84705]: from='client.64753 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:53:06.455 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:06 smithi005 ceph-mon[84705]: from='client.? 172.21.15.5:0/738498728' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T05:53:06.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:06 smithi186 ceph-mon[67903]: Upgrade: Target is version 19.0.0-2299-gea01217f (squid) 2024-03-22T05:53:06.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:06 smithi186 ceph-mon[67903]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, digests ['quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9'] 2024-03-22T05:53:06.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:06 smithi186 ceph-mon[67903]: Upgrade: Setting container_image for all mgr 2024-03-22T05:53:06.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:06 smithi186 ceph-mon[67903]: Upgrade: Setting container_image for all mon 2024-03-22T05:53:06.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:06 smithi186 ceph-mon[67903]: Upgrade: Setting container_image for all crash 2024-03-22T05:53:06.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:06 smithi186 ceph-mon[67903]: Upgrade: Setting container_image for all osd 2024-03-22T05:53:06.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:06 smithi186 ceph-mon[67903]: Upgrade: Setting container_image for all mds 2024-03-22T05:53:06.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:06 smithi186 ceph-mon[67903]: Upgrade: Setting container_image for all rgw 2024-03-22T05:53:06.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:06 smithi186 ceph-mon[67903]: Upgrade: Setting container_image for all rbd-mirror 2024-03-22T05:53:06.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:06 smithi186 ceph-mon[67903]: Upgrade: Setting container_image for all cephfs-mirror 2024-03-22T05:53:06.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:06 smithi186 ceph-mon[67903]: Upgrade: Setting container_image for all ceph-exporter 2024-03-22T05:53:06.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:06 smithi186 ceph-mon[67903]: from='client.64753 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:53:06.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:06 smithi186 ceph-mon[67903]: from='client.? 172.21.15.5:0/738498728' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T05:53:07.214 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:53:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:53:06] "GET /metrics HTTP/1.1" 200 38398 "" "Prometheus/2.43.0" 2024-03-22T05:53:07.215 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:07 smithi005 ceph-mon[89904]: pgmap v328: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 587 B/s rd, 0 op/s 2024-03-22T05:53:07.215 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:07 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:53:07.215 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:07 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:53:07.215 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:07 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:53:07.215 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:07 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:53:07.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:07 smithi005 ceph-mon[84705]: pgmap v328: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 587 B/s rd, 0 op/s 2024-03-22T05:53:07.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:07 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:53:07.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:07 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:53:07.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:07 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:53:07.215 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:07 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:53:07.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:07 smithi186 ceph-mon[67903]: pgmap v328: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 587 B/s rd, 0 op/s 2024-03-22T05:53:07.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:07 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:53:07.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:07 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:53:07.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:07 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:53:07.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:07 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:53:08.416 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:08 smithi005 ceph-mon[84705]: Upgrade: Updating iscsi.foo.smithi005.mcdwbp 2024-03-22T05:53:08.416 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:08 smithi005 ceph-mon[84705]: Deploying daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:53:08.416 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:08 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:53:08.417 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:08 smithi005 ceph-mon[89904]: Upgrade: Updating iscsi.foo.smithi005.mcdwbp 2024-03-22T05:53:08.417 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:08 smithi005 ceph-mon[89904]: Deploying daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:53:08.417 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:08 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:53:08.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:08 smithi186 ceph-mon[67903]: Upgrade: Updating iscsi.foo.smithi005.mcdwbp 2024-03-22T05:53:08.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:08 smithi186 ceph-mon[67903]: Deploying daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:53:08.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:08 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:53:09.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:09 smithi186 ceph-mon[67903]: pgmap v329: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:53:09.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:09 smithi005 ceph-mon[89904]: pgmap v329: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:53:09.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:09 smithi005 ceph-mon[84705]: pgmap v329: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:53:11.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:11 smithi186 ceph-mon[67903]: pgmap v330: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:53:11.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:11 smithi005 ceph-mon[84705]: pgmap v330: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:53:11.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:11 smithi005 ceph-mon[89904]: pgmap v330: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:53:13.504 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:13 smithi005 ceph-mon[84705]: pgmap v331: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:53:13.504 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:13 smithi005 ceph-mon[89904]: pgmap v331: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:53:13.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:13 smithi186 ceph-mon[67903]: pgmap v331: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:53:15.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:15 smithi186 ceph-mon[67903]: pgmap v332: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:53:15.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:15 smithi186 ceph-mon[67903]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:53:15.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:15 smithi005 ceph-mon[89904]: pgmap v332: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:53:15.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:15 smithi005 ceph-mon[89904]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:53:15.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:15 smithi005 ceph-mon[84705]: pgmap v332: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:53:15.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:15 smithi005 ceph-mon[84705]: from='client.15297 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:53:17.220 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:53:16 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:53:16] "GET /metrics HTTP/1.1" 200 38396 "" "Prometheus/2.43.0" 2024-03-22T05:53:17.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:17 smithi186 ceph-mon[67903]: pgmap v333: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:53:17.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:17 smithi005 ceph-mon[84705]: pgmap v333: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:53:17.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:17 smithi005 ceph-mon[89904]: pgmap v333: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:53:19.488 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:19 smithi005 ceph-mon[89904]: pgmap v334: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:53:19.488 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:19 smithi005 ceph-mon[84705]: pgmap v334: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:53:19.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:19 smithi186 ceph-mon[67903]: pgmap v334: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:53:19.908 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:53:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:53:19.560+0000 7f7f0ecc9700 -1 log_channel(cephadm) log [ERR] : Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi005.mcdwbp on host smithi005 failed. 2024-03-22T05:53:20.863 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:20 smithi005 ceph-mon[84705]: pgmap v335: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:53:20.863 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:20 smithi005 ceph-mon[84705]: Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi005.mcdwbp on host smithi005 failed. 2024-03-22T05:53:20.863 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:20 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:53:20.863 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:20 smithi005 ceph-mon[84705]: pgmap v336: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 1020 B/s rd, 0 op/s 2024-03-22T05:53:20.863 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:20 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:53:20.863 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:20 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:53:20.863 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:20 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:53:20.863 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:20 smithi005 ceph-mon[84705]: pgmap v337: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:53:20.863 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:20 smithi005 ceph-mon[84705]: pgmap v338: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 847 B/s rd, 0 op/s 2024-03-22T05:53:20.863 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:20 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:53:20.864 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:20 smithi005 ceph-mon[84705]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:53:20.864 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:20 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:53:20.864 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:20 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:53:20.864 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:20 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:53:20.864 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:20 smithi005 ceph-mon[84705]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:53:20.864 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:20 smithi005 ceph-mon[84705]: Health check failed: Upgrading daemon iscsi.foo.smithi005.mcdwbp on host smithi005 failed. (UPGRADE_REDEPLOY_DAEMON) 2024-03-22T05:53:20.864 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:20 smithi005 ceph-mon[89904]: pgmap v335: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:53:20.864 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:20 smithi005 ceph-mon[89904]: Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi005.mcdwbp on host smithi005 failed. 2024-03-22T05:53:20.864 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:20 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:53:20.864 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:20 smithi005 ceph-mon[89904]: pgmap v336: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 1020 B/s rd, 0 op/s 2024-03-22T05:53:20.864 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:20 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:53:20.864 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:20 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:53:20.865 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:20 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:53:20.865 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:20 smithi005 ceph-mon[89904]: pgmap v337: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:53:20.865 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:20 smithi005 ceph-mon[89904]: pgmap v338: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 847 B/s rd, 0 op/s 2024-03-22T05:53:20.865 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:20 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:53:20.865 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:20 smithi005 ceph-mon[89904]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:53:20.865 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:20 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:53:20.865 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:20 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:53:20.865 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:20 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:53:20.865 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:20 smithi005 ceph-mon[89904]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:53:20.865 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:20 smithi005 ceph-mon[89904]: Health check failed: Upgrading daemon iscsi.foo.smithi005.mcdwbp on host smithi005 failed. (UPGRADE_REDEPLOY_DAEMON) 2024-03-22T05:53:21.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:20 smithi186 ceph-mon[67903]: pgmap v335: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:53:21.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:20 smithi186 ceph-mon[67903]: Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi005.mcdwbp on host smithi005 failed. 2024-03-22T05:53:21.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:20 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:53:21.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:20 smithi186 ceph-mon[67903]: pgmap v336: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 1020 B/s rd, 0 op/s 2024-03-22T05:53:21.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:20 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:53:21.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:20 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:53:21.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:20 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:53:21.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:20 smithi186 ceph-mon[67903]: pgmap v337: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:53:21.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:20 smithi186 ceph-mon[67903]: pgmap v338: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 847 B/s rd, 0 op/s 2024-03-22T05:53:21.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:20 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:53:21.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:20 smithi186 ceph-mon[67903]: Reconfiguring iscsi.foo.smithi005.mcdwbp (dependencies changed)... 2024-03-22T05:53:21.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:20 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:53:21.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:20 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi005.mcdwbp", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-22T05:53:21.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:20 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:53:21.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:20 smithi186 ceph-mon[67903]: Reconfiguring daemon iscsi.foo.smithi005.mcdwbp on smithi005 2024-03-22T05:53:21.045 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:20 smithi186 ceph-mon[67903]: Health check failed: Upgrading daemon iscsi.foo.smithi005.mcdwbp on host smithi005 failed. (UPGRADE_REDEPLOY_DAEMON) 2024-03-22T05:53:23.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:23 smithi005 ceph-mon[84705]: pgmap v339: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 847 B/s rd, 0 op/s 2024-03-22T05:53:23.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:23 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:53:23.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:23 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:53:23.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:23 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-22T05:53:23.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:23 smithi005 ceph-mon[84705]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-22T05:53:23.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:23 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-22T05:53:23.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:23 smithi005 ceph-mon[84705]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-22T05:53:23.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:23 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:53:23.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:23 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:53:23.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:23 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:53:23.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:23 smithi005 ceph-mon[84705]: from='client.? 172.21.15.5:0/1970268253' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-22T05:53:23.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:23 smithi005 ceph-mon[84705]: from='client.? 172.21.15.5:0/3122723211' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6800/1985929665"}]: dispatch 2024-03-22T05:53:23.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:23 smithi005 ceph-mon[84705]: from='client.? ' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6800/1985929665"}]: dispatch 2024-03-22T05:53:23.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:23 smithi005 ceph-mon[89904]: pgmap v339: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 847 B/s rd, 0 op/s 2024-03-22T05:53:23.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:23 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:53:23.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:23 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:53:23.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:23 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-22T05:53:23.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:23 smithi005 ceph-mon[89904]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-22T05:53:23.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:23 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-22T05:53:23.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:23 smithi005 ceph-mon[89904]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-22T05:53:23.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:23 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:53:23.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:23 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:53:23.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:23 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:53:23.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:23 smithi005 ceph-mon[89904]: from='client.? 172.21.15.5:0/1970268253' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-22T05:53:23.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:23 smithi005 ceph-mon[89904]: from='client.? 172.21.15.5:0/3122723211' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6800/1985929665"}]: dispatch 2024-03-22T05:53:23.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:23 smithi005 ceph-mon[89904]: from='client.? ' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6800/1985929665"}]: dispatch 2024-03-22T05:53:23.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:23 smithi186 ceph-mon[67903]: pgmap v339: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 847 B/s rd, 0 op/s 2024-03-22T05:53:23.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:23 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:53:23.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:23 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:53:23.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:23 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-22T05:53:23.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:23 smithi186 ceph-mon[67903]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-22T05:53:23.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:23 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-22T05:53:23.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:23 smithi186 ceph-mon[67903]: from='mon.2 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-22T05:53:23.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:23 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:53:23.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:23 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:53:23.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:23 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:53:23.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:23 smithi186 ceph-mon[67903]: from='client.? 172.21.15.5:0/1970268253' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-22T05:53:23.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:23 smithi186 ceph-mon[67903]: from='client.? 172.21.15.5:0/3122723211' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6800/1985929665"}]: dispatch 2024-03-22T05:53:23.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:23 smithi186 ceph-mon[67903]: from='client.? ' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6800/1985929665"}]: dispatch 2024-03-22T05:53:24.619 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:24 smithi005 ceph-mon[89904]: from='client.? ' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6800/1985929665"}]': finished 2024-03-22T05:53:24.619 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:24 smithi005 ceph-mon[89904]: osdmap e147: 8 total, 8 up, 8 in 2024-03-22T05:53:24.619 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:24 smithi005 ceph-mon[89904]: pgmap v341: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 506 B/s rd, 0 op/s 2024-03-22T05:53:24.620 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:24 smithi005 ceph-mon[89904]: from='client.? 172.21.15.5:0/182708691' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/3200588161"}]: dispatch 2024-03-22T05:53:24.620 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:24 smithi005 ceph-mon[89904]: from='client.? ' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/3200588161"}]: dispatch 2024-03-22T05:53:24.620 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:24 smithi005 ceph-mon[84705]: from='client.? ' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6800/1985929665"}]': finished 2024-03-22T05:53:24.620 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:24 smithi005 ceph-mon[84705]: osdmap e147: 8 total, 8 up, 8 in 2024-03-22T05:53:24.620 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:24 smithi005 ceph-mon[84705]: pgmap v341: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 506 B/s rd, 0 op/s 2024-03-22T05:53:24.620 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:24 smithi005 ceph-mon[84705]: from='client.? 172.21.15.5:0/182708691' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/3200588161"}]: dispatch 2024-03-22T05:53:24.620 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:24 smithi005 ceph-mon[84705]: from='client.? ' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/3200588161"}]: dispatch 2024-03-22T05:53:24.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:24 smithi186 ceph-mon[67903]: from='client.? ' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6800/1985929665"}]': finished 2024-03-22T05:53:24.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:24 smithi186 ceph-mon[67903]: osdmap e147: 8 total, 8 up, 8 in 2024-03-22T05:53:24.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:24 smithi186 ceph-mon[67903]: pgmap v341: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 506 B/s rd, 0 op/s 2024-03-22T05:53:24.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:24 smithi186 ceph-mon[67903]: from='client.? 172.21.15.5:0/182708691' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/3200588161"}]: dispatch 2024-03-22T05:53:24.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:24 smithi186 ceph-mon[67903]: from='client.? ' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/3200588161"}]: dispatch 2024-03-22T05:53:25.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:25 smithi186 ceph-mon[67903]: from='client.? ' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/3200588161"}]': finished 2024-03-22T05:53:25.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:25 smithi186 ceph-mon[67903]: osdmap e148: 8 total, 8 up, 8 in 2024-03-22T05:53:25.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:25 smithi186 ceph-mon[67903]: from='client.? 172.21.15.5:0/1646980144' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6801/1985929665"}]: dispatch 2024-03-22T05:53:25.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:25 smithi186 ceph-mon[67903]: from='client.? ' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6801/1985929665"}]: dispatch 2024-03-22T05:53:25.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:25 smithi005 ceph-mon[84705]: from='client.? ' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/3200588161"}]': finished 2024-03-22T05:53:25.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:25 smithi005 ceph-mon[84705]: osdmap e148: 8 total, 8 up, 8 in 2024-03-22T05:53:25.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:25 smithi005 ceph-mon[84705]: from='client.? 172.21.15.5:0/1646980144' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6801/1985929665"}]: dispatch 2024-03-22T05:53:25.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:25 smithi005 ceph-mon[84705]: from='client.? ' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6801/1985929665"}]: dispatch 2024-03-22T05:53:25.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:25 smithi005 ceph-mon[89904]: from='client.? ' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/3200588161"}]': finished 2024-03-22T05:53:25.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:25 smithi005 ceph-mon[89904]: osdmap e148: 8 total, 8 up, 8 in 2024-03-22T05:53:25.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:25 smithi005 ceph-mon[89904]: from='client.? 172.21.15.5:0/1646980144' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6801/1985929665"}]: dispatch 2024-03-22T05:53:25.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:25 smithi005 ceph-mon[89904]: from='client.? ' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6801/1985929665"}]: dispatch 2024-03-22T05:53:26.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:26 smithi186 ceph-mon[67903]: from='client.? ' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6801/1985929665"}]': finished 2024-03-22T05:53:26.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:26 smithi186 ceph-mon[67903]: osdmap e149: 8 total, 8 up, 8 in 2024-03-22T05:53:26.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:26 smithi186 ceph-mon[67903]: pgmap v344: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-22T05:53:26.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:26 smithi186 ceph-mon[67903]: from='client.? 172.21.15.5:0/1271793900' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/4051704887"}]: dispatch 2024-03-22T05:53:26.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:26 smithi186 ceph-mon[67903]: from='client.? ' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/4051704887"}]: dispatch 2024-03-22T05:53:26.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:26 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:53:26.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:26 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:53:26.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:26 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:53:26.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:26 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:53:26.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:26 smithi186 ceph-mon[67903]: pgmap v345: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 463 B/s rd, 0 op/s 2024-03-22T05:53:26.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:26 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:53:26.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:26 smithi005 ceph-mon[84705]: from='client.? ' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6801/1985929665"}]': finished 2024-03-22T05:53:26.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:26 smithi005 ceph-mon[84705]: osdmap e149: 8 total, 8 up, 8 in 2024-03-22T05:53:26.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:26 smithi005 ceph-mon[84705]: pgmap v344: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-22T05:53:26.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:26 smithi005 ceph-mon[84705]: from='client.? 172.21.15.5:0/1271793900' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/4051704887"}]: dispatch 2024-03-22T05:53:26.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:26 smithi005 ceph-mon[84705]: from='client.? ' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/4051704887"}]: dispatch 2024-03-22T05:53:26.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:26 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:53:26.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:26 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:53:26.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:26 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:53:26.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:26 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:53:26.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:26 smithi005 ceph-mon[84705]: pgmap v345: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 463 B/s rd, 0 op/s 2024-03-22T05:53:26.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:26 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:53:26.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:26 smithi005 ceph-mon[89904]: from='client.? ' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6801/1985929665"}]': finished 2024-03-22T05:53:26.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:26 smithi005 ceph-mon[89904]: osdmap e149: 8 total, 8 up, 8 in 2024-03-22T05:53:26.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:26 smithi005 ceph-mon[89904]: pgmap v344: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-22T05:53:26.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:26 smithi005 ceph-mon[89904]: from='client.? 172.21.15.5:0/1271793900' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/4051704887"}]: dispatch 2024-03-22T05:53:26.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:26 smithi005 ceph-mon[89904]: from='client.? ' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/4051704887"}]: dispatch 2024-03-22T05:53:26.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:26 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:53:26.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:26 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:53:26.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:26 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:53:26.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:26 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:53:26.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:26 smithi005 ceph-mon[89904]: pgmap v345: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 463 B/s rd, 0 op/s 2024-03-22T05:53:26.954 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:26 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:53:27.452 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:53:26 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:53:26] "GET /metrics HTTP/1.1" 200 38473 "" "Prometheus/2.43.0" 2024-03-22T05:53:27.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:27 smithi186 ceph-mon[67903]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-22T05:53:27.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:27 smithi186 ceph-mon[67903]: from='client.? ' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/4051704887"}]': finished 2024-03-22T05:53:27.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:27 smithi186 ceph-mon[67903]: osdmap e150: 8 total, 8 up, 8 in 2024-03-22T05:53:27.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:27 smithi186 ceph-mon[67903]: from='client.? 172.21.15.5:0/3526727607' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/4239998749"}]: dispatch 2024-03-22T05:53:27.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:27 smithi186 ceph-mon[67903]: from='client.? ' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/4239998749"}]: dispatch 2024-03-22T05:53:27.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:27 smithi005 ceph-mon[84705]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-22T05:53:27.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:27 smithi005 ceph-mon[84705]: from='client.? ' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/4051704887"}]': finished 2024-03-22T05:53:27.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:27 smithi005 ceph-mon[84705]: osdmap e150: 8 total, 8 up, 8 in 2024-03-22T05:53:27.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:27 smithi005 ceph-mon[84705]: from='client.? 172.21.15.5:0/3526727607' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/4239998749"}]: dispatch 2024-03-22T05:53:27.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:27 smithi005 ceph-mon[84705]: from='client.? ' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/4239998749"}]: dispatch 2024-03-22T05:53:27.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:27 smithi005 ceph-mon[89904]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-22T05:53:27.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:27 smithi005 ceph-mon[89904]: from='client.? ' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/4051704887"}]': finished 2024-03-22T05:53:27.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:27 smithi005 ceph-mon[89904]: osdmap e150: 8 total, 8 up, 8 in 2024-03-22T05:53:27.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:27 smithi005 ceph-mon[89904]: from='client.? 172.21.15.5:0/3526727607' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/4239998749"}]: dispatch 2024-03-22T05:53:27.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:27 smithi005 ceph-mon[89904]: from='client.? ' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/4239998749"}]: dispatch 2024-03-22T05:53:28.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:28 smithi005 ceph-mon[84705]: from='client.? ' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/4239998749"}]': finished 2024-03-22T05:53:28.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:28 smithi005 ceph-mon[84705]: osdmap e151: 8 total, 8 up, 8 in 2024-03-22T05:53:28.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:28 smithi005 ceph-mon[84705]: from='client.? 172.21.15.5:0/681027493' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/932763527"}]: dispatch 2024-03-22T05:53:28.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:28 smithi005 ceph-mon[84705]: pgmap v348: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-22T05:53:28.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:28 smithi005 ceph-mon[89904]: from='client.? ' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/4239998749"}]': finished 2024-03-22T05:53:28.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:28 smithi005 ceph-mon[89904]: osdmap e151: 8 total, 8 up, 8 in 2024-03-22T05:53:28.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:28 smithi005 ceph-mon[89904]: from='client.? 172.21.15.5:0/681027493' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/932763527"}]: dispatch 2024-03-22T05:53:28.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:28 smithi005 ceph-mon[89904]: pgmap v348: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-22T05:53:29.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:28 smithi186 ceph-mon[67903]: from='client.? ' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/4239998749"}]': finished 2024-03-22T05:53:29.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:28 smithi186 ceph-mon[67903]: osdmap e151: 8 total, 8 up, 8 in 2024-03-22T05:53:29.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:28 smithi186 ceph-mon[67903]: from='client.? 172.21.15.5:0/681027493' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/932763527"}]: dispatch 2024-03-22T05:53:29.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:28 smithi186 ceph-mon[67903]: pgmap v348: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-22T05:53:29.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:29 smithi005 ceph-mon[84705]: from='client.? 172.21.15.5:0/681027493' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/932763527"}]': finished 2024-03-22T05:53:29.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:29 smithi005 ceph-mon[84705]: osdmap e152: 8 total, 8 up, 8 in 2024-03-22T05:53:29.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:29 smithi005 ceph-mon[84705]: from='client.? 172.21.15.5:0/4115739264' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6800/3755200857"}]: dispatch 2024-03-22T05:53:29.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:29 smithi005 ceph-mon[84705]: from='client.? ' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6800/3755200857"}]: dispatch 2024-03-22T05:53:29.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:29 smithi005 ceph-mon[89904]: from='client.? 172.21.15.5:0/681027493' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/932763527"}]': finished 2024-03-22T05:53:29.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:29 smithi005 ceph-mon[89904]: osdmap e152: 8 total, 8 up, 8 in 2024-03-22T05:53:29.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:29 smithi005 ceph-mon[89904]: from='client.? 172.21.15.5:0/4115739264' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6800/3755200857"}]: dispatch 2024-03-22T05:53:29.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:29 smithi005 ceph-mon[89904]: from='client.? ' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6800/3755200857"}]: dispatch 2024-03-22T05:53:30.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:29 smithi186 ceph-mon[67903]: from='client.? 172.21.15.5:0/681027493' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/932763527"}]': finished 2024-03-22T05:53:30.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:29 smithi186 ceph-mon[67903]: osdmap e152: 8 total, 8 up, 8 in 2024-03-22T05:53:30.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:29 smithi186 ceph-mon[67903]: from='client.? 172.21.15.5:0/4115739264' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6800/3755200857"}]: dispatch 2024-03-22T05:53:30.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:29 smithi186 ceph-mon[67903]: from='client.? ' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6800/3755200857"}]: dispatch 2024-03-22T05:53:30.945 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:30 smithi186 ceph-mon[67903]: from='client.? ' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6800/3755200857"}]': finished 2024-03-22T05:53:30.945 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:30 smithi186 ceph-mon[67903]: osdmap e153: 8 total, 8 up, 8 in 2024-03-22T05:53:30.946 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:30 smithi186 ceph-mon[67903]: from='client.? 172.21.15.5:0/2138292886' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/529385790"}]: dispatch 2024-03-22T05:53:30.946 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:30 smithi186 ceph-mon[67903]: pgmap v351: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:53:30.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:30 smithi005 ceph-mon[84705]: from='client.? ' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6800/3755200857"}]': finished 2024-03-22T05:53:30.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:30 smithi005 ceph-mon[84705]: osdmap e153: 8 total, 8 up, 8 in 2024-03-22T05:53:30.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:30 smithi005 ceph-mon[84705]: from='client.? 172.21.15.5:0/2138292886' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/529385790"}]: dispatch 2024-03-22T05:53:30.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:30 smithi005 ceph-mon[84705]: pgmap v351: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:53:30.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:30 smithi005 ceph-mon[89904]: from='client.? ' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6800/3755200857"}]': finished 2024-03-22T05:53:30.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:30 smithi005 ceph-mon[89904]: osdmap e153: 8 total, 8 up, 8 in 2024-03-22T05:53:30.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:30 smithi005 ceph-mon[89904]: from='client.? 172.21.15.5:0/2138292886' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/529385790"}]: dispatch 2024-03-22T05:53:30.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:30 smithi005 ceph-mon[89904]: pgmap v351: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:53:31.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:31 smithi005 ceph-mon[84705]: from='client.? 172.21.15.5:0/2138292886' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/529385790"}]': finished 2024-03-22T05:53:31.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:31 smithi005 ceph-mon[84705]: osdmap e154: 8 total, 8 up, 8 in 2024-03-22T05:53:31.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:31 smithi005 ceph-mon[84705]: from='client.? 172.21.15.5:0/4241073549' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/2085560383"}]: dispatch 2024-03-22T05:53:31.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:31 smithi005 ceph-mon[89904]: from='client.? 172.21.15.5:0/2138292886' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/529385790"}]': finished 2024-03-22T05:53:31.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:31 smithi005 ceph-mon[89904]: osdmap e154: 8 total, 8 up, 8 in 2024-03-22T05:53:31.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:31 smithi005 ceph-mon[89904]: from='client.? 172.21.15.5:0/4241073549' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/2085560383"}]: dispatch 2024-03-22T05:53:32.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:31 smithi186 ceph-mon[67903]: from='client.? 172.21.15.5:0/2138292886' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/529385790"}]': finished 2024-03-22T05:53:32.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:31 smithi186 ceph-mon[67903]: osdmap e154: 8 total, 8 up, 8 in 2024-03-22T05:53:32.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:31 smithi186 ceph-mon[67903]: from='client.? 172.21.15.5:0/4241073549' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/2085560383"}]: dispatch 2024-03-22T05:53:32.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:32 smithi005 ceph-mon[89904]: from='client.? 172.21.15.5:0/4241073549' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/2085560383"}]': finished 2024-03-22T05:53:32.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:32 smithi005 ceph-mon[89904]: osdmap e155: 8 total, 8 up, 8 in 2024-03-22T05:53:32.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:32 smithi005 ceph-mon[89904]: from='client.? 172.21.15.5:0/3014283511' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6801/3755200857"}]: dispatch 2024-03-22T05:53:32.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:32 smithi005 ceph-mon[89904]: pgmap v354: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:53:32.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:32 smithi005 ceph-mon[89904]: from='client.44657 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:53:32.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:32 smithi005 ceph-mon[84705]: from='client.? 172.21.15.5:0/4241073549' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/2085560383"}]': finished 2024-03-22T05:53:32.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:32 smithi005 ceph-mon[84705]: osdmap e155: 8 total, 8 up, 8 in 2024-03-22T05:53:32.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:32 smithi005 ceph-mon[84705]: from='client.? 172.21.15.5:0/3014283511' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6801/3755200857"}]: dispatch 2024-03-22T05:53:32.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:32 smithi005 ceph-mon[84705]: pgmap v354: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:53:32.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:32 smithi005 ceph-mon[84705]: from='client.44657 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:53:33.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:32 smithi186 ceph-mon[67903]: from='client.? 172.21.15.5:0/4241073549' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/2085560383"}]': finished 2024-03-22T05:53:33.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:32 smithi186 ceph-mon[67903]: osdmap e155: 8 total, 8 up, 8 in 2024-03-22T05:53:33.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:32 smithi186 ceph-mon[67903]: from='client.? 172.21.15.5:0/3014283511' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6801/3755200857"}]: dispatch 2024-03-22T05:53:33.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:32 smithi186 ceph-mon[67903]: pgmap v354: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:53:33.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:32 smithi186 ceph-mon[67903]: from='client.44657 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:53:33.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:33 smithi005 ceph-mon[89904]: from='client.? 172.21.15.5:0/3014283511' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6801/3755200857"}]': finished 2024-03-22T05:53:33.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:33 smithi005 ceph-mon[89904]: osdmap e156: 8 total, 8 up, 8 in 2024-03-22T05:53:33.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:33 smithi005 ceph-mon[89904]: from='client.? 172.21.15.5:0/2915643009' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/4045119591"}]: dispatch 2024-03-22T05:53:33.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:33 smithi005 ceph-mon[89904]: from='client.? ' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/4045119591"}]: dispatch 2024-03-22T05:53:33.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:33 smithi005 ceph-mon[84705]: from='client.? 172.21.15.5:0/3014283511' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6801/3755200857"}]': finished 2024-03-22T05:53:33.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:33 smithi005 ceph-mon[84705]: osdmap e156: 8 total, 8 up, 8 in 2024-03-22T05:53:33.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:33 smithi005 ceph-mon[84705]: from='client.? 172.21.15.5:0/2915643009' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/4045119591"}]: dispatch 2024-03-22T05:53:33.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:33 smithi005 ceph-mon[84705]: from='client.? ' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/4045119591"}]: dispatch 2024-03-22T05:53:34.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:33 smithi186 ceph-mon[67903]: from='client.? 172.21.15.5:0/3014283511' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:6801/3755200857"}]': finished 2024-03-22T05:53:34.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:33 smithi186 ceph-mon[67903]: osdmap e156: 8 total, 8 up, 8 in 2024-03-22T05:53:34.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:33 smithi186 ceph-mon[67903]: from='client.? 172.21.15.5:0/2915643009' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/4045119591"}]: dispatch 2024-03-22T05:53:34.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:33 smithi186 ceph-mon[67903]: from='client.? ' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/4045119591"}]: dispatch 2024-03-22T05:53:34.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:34 smithi005 ceph-mon[84705]: from='client.? ' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/4045119591"}]': finished 2024-03-22T05:53:34.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:34 smithi005 ceph-mon[84705]: osdmap e157: 8 total, 8 up, 8 in 2024-03-22T05:53:34.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:34 smithi005 ceph-mon[84705]: from='client.? 172.21.15.5:0/3304509490' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/3242396524"}]: dispatch 2024-03-22T05:53:34.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:34 smithi005 ceph-mon[84705]: pgmap v357: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:53:34.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:34 smithi005 ceph-mon[89904]: from='client.? ' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/4045119591"}]': finished 2024-03-22T05:53:34.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:34 smithi005 ceph-mon[89904]: osdmap e157: 8 total, 8 up, 8 in 2024-03-22T05:53:34.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:34 smithi005 ceph-mon[89904]: from='client.? 172.21.15.5:0/3304509490' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/3242396524"}]: dispatch 2024-03-22T05:53:34.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:34 smithi005 ceph-mon[89904]: pgmap v357: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:53:35.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:34 smithi186 ceph-mon[67903]: from='client.? ' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/4045119591"}]': finished 2024-03-22T05:53:35.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:34 smithi186 ceph-mon[67903]: osdmap e157: 8 total, 8 up, 8 in 2024-03-22T05:53:35.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:34 smithi186 ceph-mon[67903]: from='client.? 172.21.15.5:0/3304509490' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/3242396524"}]: dispatch 2024-03-22T05:53:35.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:34 smithi186 ceph-mon[67903]: pgmap v357: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:53:35.743 INFO:teuthology.orchestra.run.smithi005.stdout:true 2024-03-22T05:53:35.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:35 smithi005 ceph-mon[84705]: from='client.? 172.21.15.5:0/3304509490' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/3242396524"}]': finished 2024-03-22T05:53:35.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:35 smithi005 ceph-mon[84705]: osdmap e158: 8 total, 8 up, 8 in 2024-03-22T05:53:35.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:35 smithi005 ceph-mon[89904]: from='client.? 172.21.15.5:0/3304509490' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/3242396524"}]': finished 2024-03-22T05:53:35.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:35 smithi005 ceph-mon[89904]: osdmap e158: 8 total, 8 up, 8 in 2024-03-22T05:53:36.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:35 smithi186 ceph-mon[67903]: from='client.? 172.21.15.5:0/3304509490' entity='client.iscsi.foo.smithi005.mcdwbp' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.5:0/3242396524"}]': finished 2024-03-22T05:53:36.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:35 smithi186 ceph-mon[67903]: osdmap e158: 8 total, 8 up, 8 in 2024-03-22T05:53:36.094 INFO:teuthology.orchestra.run.smithi005.stdout:"Error: UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi005.mcdwbp on host smithi005 failed." 2024-03-22T05:53:36.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:36 smithi005 ceph-mon[89904]: from='client.54936 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:53:36.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:36 smithi005 ceph-mon[89904]: pgmap v359: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:53:36.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:36 smithi005 ceph-mon[89904]: from='client.64867 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:53:36.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:36 smithi005 ceph-mon[84705]: from='client.54936 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:53:36.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:36 smithi005 ceph-mon[84705]: pgmap v359: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:53:36.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:36 smithi005 ceph-mon[84705]: from='client.64867 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:53:36.926 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph orch ps' 2024-03-22T05:53:37.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:36 smithi186 ceph-mon[67903]: from='client.54936 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:53:37.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:36 smithi186 ceph-mon[67903]: pgmap v359: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-22T05:53:37.044 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:36 smithi186 ceph-mon[67903]: from='client.64867 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:53:37.202 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:53:36 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:53:36] "GET /metrics HTTP/1.1" 200 38473 "" "Prometheus/2.43.0" 2024-03-22T05:53:38.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:38 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:53:38.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:38 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:53:38.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:38 smithi186 ceph-mon[67903]: pgmap v360: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:53:38.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:38 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:53:38.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:38 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:53:38.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:38 smithi005 ceph-mon[84705]: pgmap v360: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:53:38.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:38 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:53:38.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:38 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:53:38.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:38 smithi005 ceph-mon[89904]: pgmap v360: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:53:39.007 INFO:teuthology.orchestra.run.smithi005.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T05:53:39.007 INFO:teuthology.orchestra.run.smithi005.stdout:alertmanager.a smithi005 *:9093,9094 running (11m) 13s ago 33m - - 0.25.0 c8568f914cd2 ee99dfc435e8 2024-03-22T05:53:39.007 INFO:teuthology.orchestra.run.smithi005.stdout:grafana.a smithi186 *:3000 running (10m) 71s ago 32m 83.5M - 9.4.12 2bacad6d85d8 16fd57dbbd71 2024-03-22T05:53:39.007 INFO:teuthology.orchestra.run.smithi005.stdout:iscsi.foo.smithi005.mcdwbp smithi005 running (17s) 13s ago 32m - - 3.7 aa461e7c5e7f 8a50f2d11e66 2024-03-22T05:53:39.007 INFO:teuthology.orchestra.run.smithi005.stdout:mgr.x smithi186 *:8443,9283,8765 running (12m) 71s ago 37m 454M - 19.0.0-2299-gea01217f aa461e7c5e7f c556421a73f3 2024-03-22T05:53:39.007 INFO:teuthology.orchestra.run.smithi005.stdout:mgr.y smithi005 *:8443,9283,8765 running (22m) 13s ago 39m - - 19.0.0-2299-gea01217f aa461e7c5e7f 62a813eebc0d 2024-03-22T05:53:39.007 INFO:teuthology.orchestra.run.smithi005.stdout:mon.a smithi005 running (8m) 13s ago 39m - 2048M 19.0.0-2299-gea01217f aa461e7c5e7f 111a2aa24dd9 2024-03-22T05:53:39.008 INFO:teuthology.orchestra.run.smithi005.stdout:mon.b smithi186 running (9m) 71s ago 37m 55.8M 2048M 19.0.0-2299-gea01217f aa461e7c5e7f c1798feac15b 2024-03-22T05:53:39.008 INFO:teuthology.orchestra.run.smithi005.stdout:mon.c smithi005 running (8m) 13s ago 37m - 2048M 19.0.0-2299-gea01217f aa461e7c5e7f 708c72ff74f5 2024-03-22T05:53:39.008 INFO:teuthology.orchestra.run.smithi005.stdout:node-exporter.a smithi005 *:9100 running (11m) 13s ago 33m - - 1.5.0 0da6a335fe13 f635cd72b9d3 2024-03-22T05:53:39.008 INFO:teuthology.orchestra.run.smithi005.stdout:node-exporter.b smithi186 *:9100 running (11m) 71s ago 33m 14.7M - 1.5.0 0da6a335fe13 d7e0b3d20191 2024-03-22T05:53:39.008 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0 smithi005 running (6m) 13s ago 37m - 2993M 19.0.0-2299-gea01217f aa461e7c5e7f 31b6cdee5f5b 2024-03-22T05:53:39.008 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1 smithi005 running (6m) 13s ago 36m - 2993M 19.0.0-2299-gea01217f aa461e7c5e7f 7d21bc9fd3c2 2024-03-22T05:53:39.008 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2 smithi005 running (5m) 13s ago 36m - 2993M 19.0.0-2299-gea01217f aa461e7c5e7f 628204cb7c98 2024-03-22T05:53:39.008 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3 smithi005 running (4m) 13s ago 36m - 2993M 19.0.0-2299-gea01217f aa461e7c5e7f d31a01a0230c 2024-03-22T05:53:39.008 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4 smithi186 running (3m) 71s ago 35m 63.4M 3248M 19.0.0-2299-gea01217f aa461e7c5e7f 8f7619b726f3 2024-03-22T05:53:39.008 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5 smithi186 running (3m) 71s ago 35m 53.8M 3248M 19.0.0-2299-gea01217f aa461e7c5e7f b80f7eceb2d7 2024-03-22T05:53:39.008 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6 smithi186 running (2m) 71s ago 34m 55.5M 3248M 19.0.0-2299-gea01217f aa461e7c5e7f db7c81962c65 2024-03-22T05:53:39.008 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7 smithi186 running (2m) 71s ago 34m 50.0M 3248M 19.0.0-2299-gea01217f aa461e7c5e7f fcb46ba843ae 2024-03-22T05:53:39.008 INFO:teuthology.orchestra.run.smithi005.stdout:prometheus.a smithi186 *:9095 running (11m) 71s ago 33m 72.0M - 2.43.0 a07b618ecd1d a5a043588d6d 2024-03-22T05:53:39.008 INFO:teuthology.orchestra.run.smithi005.stdout:rgw.foo.smithi005.exdhqu smithi005 *:8000 running (81s) 13s ago 32m - - 19.0.0-2299-gea01217f aa461e7c5e7f aa3dec0ece5c 2024-03-22T05:53:39.008 INFO:teuthology.orchestra.run.smithi005.stdout:rgw.foo.smithi186.mcgodk smithi186 *:8000 running (75s) 71s ago 32m 75.9M - 19.0.0-2299-gea01217f aa461e7c5e7f c7967b92514a 2024-03-22T05:53:39.521 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:39 smithi005 ceph-mon[84705]: from='client.54948 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:53:39.643 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph versions' 2024-03-22T05:53:39.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:39 smithi186 ceph-mon[67903]: from='client.54948 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:53:39.952 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:39 smithi005 ceph-mon[89904]: from='client.54948 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:53:40.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:40 smithi186 ceph-mon[67903]: pgmap v361: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 690 B/s rd, 0 op/s 2024-03-22T05:53:40.953 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:40 smithi005 ceph-mon[84705]: pgmap v361: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 690 B/s rd, 0 op/s 2024-03-22T05:53:40.953 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:40 smithi005 ceph-mon[89904]: pgmap v361: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 690 B/s rd, 0 op/s 2024-03-22T05:53:41.875 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-03-22T05:53:41.875 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": { 2024-03-22T05:53:41.875 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev)": 3 2024-03-22T05:53:41.876 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:53:41.876 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr": { 2024-03-22T05:53:41.876 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev)": 2 2024-03-22T05:53:41.876 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:53:41.876 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": { 2024-03-22T05:53:41.876 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev)": 8 2024-03-22T05:53:41.876 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:53:41.876 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw": { 2024-03-22T05:53:41.876 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev)": 2 2024-03-22T05:53:41.876 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:53:41.876 INFO:teuthology.orchestra.run.smithi005.stdout: "overall": { 2024-03-22T05:53:41.876 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev)": 15 2024-03-22T05:53:41.876 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-03-22T05:53:41.876 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-03-22T05:53:42.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:41 smithi005 ceph-mon[89904]: from='client.? 172.21.15.5:0/2558885990' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:53:42.202 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:41 smithi005 ceph-mon[84705]: from='client.? 172.21.15.5:0/2558885990' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:53:42.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:41 smithi186 ceph-mon[67903]: from='client.? 172.21.15.5:0/2558885990' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:53:42.864 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'echo "wait for servicemap items w/ changing names to refresh"' 2024-03-22T05:53:43.182 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:42 smithi005 ceph-mon[84705]: pgmap v362: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:53:43.183 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:42 smithi005 ceph-mon[84705]: from='client.44657 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:53:43.183 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:42 smithi005 ceph-mon[89904]: pgmap v362: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:53:43.183 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:42 smithi005 ceph-mon[89904]: from='client.44657 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:53:43.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:42 smithi186 ceph-mon[67903]: pgmap v362: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:53:43.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:42 smithi186 ceph-mon[67903]: from='client.44657 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:53:44.622 INFO:teuthology.orchestra.run.smithi005.stdout:wait for servicemap items w/ changing names to refresh 2024-03-22T05:53:45.376 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:45 smithi005 ceph-mon[84705]: pgmap v363: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:53:45.376 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:45 smithi005 ceph-mon[89904]: pgmap v363: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:53:45.405 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'sleep 60' 2024-03-22T05:53:45.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:45 smithi186 ceph-mon[67903]: pgmap v363: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-22T05:53:47.047 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:53:46 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:53:46] "GET /metrics HTTP/1.1" 200 38477 "" "Prometheus/2.43.0" 2024-03-22T05:53:47.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:47 smithi005 ceph-mon[84705]: pgmap v364: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 897 B/s rd, 0 op/s 2024-03-22T05:53:47.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:47 smithi005 ceph-mon[89904]: pgmap v364: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 897 B/s rd, 0 op/s 2024-03-22T05:53:47.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:47 smithi186 ceph-mon[67903]: pgmap v364: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 897 B/s rd, 0 op/s 2024-03-22T05:53:49.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:49 smithi005 ceph-mon[89904]: pgmap v365: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:53:49.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:49 smithi005 ceph-mon[84705]: pgmap v365: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:53:49.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:49 smithi186 ceph-mon[67903]: pgmap v365: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:53:51.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:51 smithi005 ceph-mon[89904]: pgmap v366: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:53:51.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:51 smithi005 ceph-mon[84705]: pgmap v366: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:53:51.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:51 smithi186 ceph-mon[67903]: pgmap v366: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:53:53.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:53 smithi005 ceph-mon[84705]: pgmap v367: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:53:53.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:53 smithi005 ceph-mon[84705]: from='client.44657 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:53:53.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:53 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:53:53.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:53 smithi005 ceph-mon[89904]: pgmap v367: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:53:53.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:53 smithi005 ceph-mon[89904]: from='client.44657 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:53:53.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:53 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:53:53.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:53 smithi186 ceph-mon[67903]: pgmap v367: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:53:53.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:53 smithi186 ceph-mon[67903]: from='client.44657 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:53:53.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:53 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:53:55.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:55 smithi005 ceph-mon[84705]: pgmap v368: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:53:55.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:55 smithi005 ceph-mon[89904]: pgmap v368: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:53:55.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:55 smithi186 ceph-mon[67903]: pgmap v368: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:53:57.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:57 smithi005 ceph-mon[84705]: pgmap v369: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:53:57.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:57 smithi005 ceph-mon[89904]: pgmap v369: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:53:57.453 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:53:56 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:53:56] "GET /metrics HTTP/1.1" 200 38473 "" "Prometheus/2.43.0" 2024-03-22T05:53:57.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:57 smithi186 ceph-mon[67903]: pgmap v369: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:53:59.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:53:59 smithi005 ceph-mon[89904]: pgmap v370: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:53:59.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:53:59 smithi005 ceph-mon[84705]: pgmap v370: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:53:59.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:53:59 smithi186 ceph-mon[67903]: pgmap v370: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:54:01.346 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:54:01 smithi186 ceph-mon[67903]: pgmap v371: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:54:01.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:54:01 smithi005 ceph-mon[84705]: pgmap v371: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:54:01.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:54:01 smithi005 ceph-mon[89904]: pgmap v371: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:54:03.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:54:03 smithi005 ceph-mon[84705]: pgmap v372: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:54:03.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:54:03 smithi005 ceph-mon[84705]: from='client.44657 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:54:03.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:54:03 smithi005 ceph-mon[89904]: pgmap v372: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:54:03.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:54:03 smithi005 ceph-mon[89904]: from='client.44657 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:54:03.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:54:03 smithi186 ceph-mon[67903]: pgmap v372: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:54:03.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:54:03 smithi186 ceph-mon[67903]: from='client.44657 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:54:05.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:54:05 smithi005 ceph-mon[84705]: pgmap v373: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:54:05.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:54:05 smithi005 ceph-mon[89904]: pgmap v373: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:54:05.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:54:05 smithi186 ceph-mon[67903]: pgmap v373: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:54:07.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:54:07 smithi005 ceph-mon[84705]: pgmap v374: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:54:07.453 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:54:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:54:06] "GET /metrics HTTP/1.1" 200 38473 "" "Prometheus/2.43.0" 2024-03-22T05:54:07.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:54:07 smithi005 ceph-mon[89904]: pgmap v374: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:54:07.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:54:07 smithi186 ceph-mon[67903]: pgmap v374: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:54:08.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:54:08 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:54:08.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:54:08 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:54:08.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:54:08 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:54:09.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:54:09 smithi005 ceph-mon[84705]: pgmap v375: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:54:09.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:54:09 smithi005 ceph-mon[89904]: pgmap v375: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:54:09.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:54:09 smithi186 ceph-mon[67903]: pgmap v375: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:54:11.454 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:54:11 smithi005 ceph-mon[84705]: pgmap v376: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:54:11.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:54:11 smithi005 ceph-mon[89904]: pgmap v376: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:54:11.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:54:11 smithi186 ceph-mon[67903]: pgmap v376: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:54:13.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:54:13 smithi005 ceph-mon[89904]: pgmap v377: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:54:13.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:54:13 smithi005 ceph-mon[89904]: from='client.44657 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:54:13.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:54:13 smithi005 ceph-mon[84705]: pgmap v377: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:54:13.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:54:13 smithi005 ceph-mon[84705]: from='client.44657 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:54:13.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:54:13 smithi186 ceph-mon[67903]: pgmap v377: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:54:13.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:54:13 smithi186 ceph-mon[67903]: from='client.44657 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:54:15.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:54:15 smithi005 ceph-mon[89904]: pgmap v378: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:54:15.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:54:15 smithi005 ceph-mon[84705]: pgmap v378: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:54:15.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:54:15 smithi186 ceph-mon[67903]: pgmap v378: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:54:17.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:54:17 smithi005 ceph-mon[84705]: pgmap v379: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:54:17.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:54:17 smithi005 ceph-mon[89904]: pgmap v379: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:54:17.453 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:54:16 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:54:16] "GET /metrics HTTP/1.1" 200 38478 "" "Prometheus/2.43.0" 2024-03-22T05:54:17.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:54:17 smithi186 ceph-mon[67903]: pgmap v379: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:54:19.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:54:19 smithi005 ceph-mon[84705]: pgmap v380: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:54:19.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:54:19 smithi005 ceph-mon[89904]: pgmap v380: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:54:19.543 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:54:19 smithi186 ceph-mon[67903]: pgmap v380: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:54:21.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:54:21 smithi005 ceph-mon[89904]: pgmap v381: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:54:21.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:54:21 smithi005 ceph-mon[84705]: pgmap v381: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:54:21.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:54:21 smithi186 ceph-mon[67903]: pgmap v381: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:54:23.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:54:23 smithi005 ceph-mon[84705]: pgmap v382: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:54:23.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:54:23 smithi005 ceph-mon[84705]: from='client.44657 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:54:23.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:54:23 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:54:23.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:54:23 smithi005 ceph-mon[89904]: pgmap v382: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:54:23.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:54:23 smithi005 ceph-mon[89904]: from='client.44657 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:54:23.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:54:23 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:54:23.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:54:23 smithi186 ceph-mon[67903]: pgmap v382: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:54:23.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:54:23 smithi186 ceph-mon[67903]: from='client.44657 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:54:23.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:54:23 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:54:25.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:54:25 smithi005 ceph-mon[84705]: pgmap v383: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:54:25.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:54:25 smithi005 ceph-mon[89904]: pgmap v383: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:54:25.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:54:25 smithi186 ceph-mon[67903]: pgmap v383: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:54:27.452 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:54:26 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:54:26] "GET /metrics HTTP/1.1" 200 38470 "" "Prometheus/2.43.0" 2024-03-22T05:54:27.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:54:27 smithi005 ceph-mon[84705]: pgmap v384: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:54:27.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:54:27 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:54:27.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:54:27 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:54:27.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:54:27 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:54:27.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:54:27 smithi005 ceph-mon[84705]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:54:27.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:54:27 smithi005 ceph-mon[89904]: pgmap v384: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:54:27.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:54:27 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:54:27.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:54:27 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:54:27.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:54:27 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:54:27.454 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:54:27 smithi005 ceph-mon[89904]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:54:27.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:54:27 smithi186 ceph-mon[67903]: pgmap v384: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:54:27.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:54:27 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-22T05:54:27.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:54:27 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-22T05:54:27.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:54:27 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-22T05:54:27.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:54:27 smithi186 ceph-mon[67903]: from='mgr.54108 ' entity='mgr.y' 2024-03-22T05:54:29.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:54:29 smithi005 ceph-mon[84705]: pgmap v385: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:54:29.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:54:29 smithi005 ceph-mon[89904]: pgmap v385: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:54:29.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:54:29 smithi186 ceph-mon[67903]: pgmap v385: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:54:31.596 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:54:31 smithi186 ceph-mon[67903]: pgmap v386: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:54:31.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:54:31 smithi005 ceph-mon[84705]: pgmap v386: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:54:31.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:54:31 smithi005 ceph-mon[89904]: pgmap v386: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:54:33.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:54:33 smithi005 ceph-mon[84705]: pgmap v387: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:54:33.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:54:33 smithi005 ceph-mon[84705]: from='client.44657 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:54:33.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:54:33 smithi005 ceph-mon[89904]: pgmap v387: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:54:33.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:54:33 smithi005 ceph-mon[89904]: from='client.44657 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:54:33.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:54:33 smithi186 ceph-mon[67903]: pgmap v387: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:54:33.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:54:33 smithi186 ceph-mon[67903]: from='client.44657 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:54:35.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:54:35 smithi005 ceph-mon[84705]: pgmap v388: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:54:35.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:54:35 smithi005 ceph-mon[89904]: pgmap v388: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:54:35.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:54:35 smithi186 ceph-mon[67903]: pgmap v388: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:54:37.304 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:54:36 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:54:36] "GET /metrics HTTP/1.1" 200 38470 "" "Prometheus/2.43.0" 2024-03-22T05:54:37.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:54:37 smithi005 ceph-mon[89904]: pgmap v389: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:54:37.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:54:37 smithi005 ceph-mon[84705]: pgmap v389: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:54:37.793 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:54:37 smithi186 ceph-mon[67903]: pgmap v389: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:54:38.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:54:38 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:54:38.703 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:54:38 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:54:38.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:54:38 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:54:39.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:54:39 smithi005 ceph-mon[84705]: pgmap v390: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:54:39.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:54:39 smithi005 ceph-mon[89904]: pgmap v390: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:54:39.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:54:39 smithi186 ceph-mon[67903]: pgmap v390: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:54:41.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:54:41 smithi005 ceph-mon[84705]: pgmap v391: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:54:41.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:54:41 smithi005 ceph-mon[89904]: pgmap v391: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:54:41.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:54:41 smithi186 ceph-mon[67903]: pgmap v391: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:54:43.604 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:54:43 smithi005 ceph-mon[84705]: pgmap v392: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:54:43.604 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:54:43 smithi005 ceph-mon[84705]: from='client.44657 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:54:43.604 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:54:43 smithi005 ceph-mon[89904]: pgmap v392: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:54:43.604 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:54:43 smithi005 ceph-mon[89904]: from='client.44657 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:54:43.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:54:43 smithi186 ceph-mon[67903]: pgmap v392: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:54:43.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:54:43 smithi186 ceph-mon[67903]: from='client.44657 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:54:45.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:54:45 smithi005 ceph-mon[84705]: pgmap v393: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:54:45.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:54:45 smithi005 ceph-mon[89904]: pgmap v393: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:54:45.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:54:45 smithi186 ceph-mon[67903]: pgmap v393: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:54:47.169 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:54:46 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:54:46] "GET /metrics HTTP/1.1" 200 38474 "" "Prometheus/2.43.0" 2024-03-22T05:54:47.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:54:47 smithi005 ceph-mon[84705]: pgmap v394: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:54:47.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:54:47 smithi005 ceph-mon[89904]: pgmap v394: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:54:47.585 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph orch ps' 2024-03-22T05:54:47.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:54:47 smithi186 ceph-mon[67903]: pgmap v394: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:54:49.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:54:49 smithi005 ceph-mon[84705]: pgmap v395: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:54:49.452 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:54:49 smithi005 ceph-mon[89904]: pgmap v395: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:54:49.656 INFO:teuthology.orchestra.run.smithi005.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-22T05:54:49.656 INFO:teuthology.orchestra.run.smithi005.stdout:alertmanager.a smithi005 *:9093,9094 running (12m) 83s ago 34m - - 0.25.0 c8568f914cd2 ee99dfc435e8 2024-03-22T05:54:49.656 INFO:teuthology.orchestra.run.smithi005.stdout:grafana.a smithi186 *:3000 running (12m) 2m ago 34m 83.5M - 9.4.12 2bacad6d85d8 16fd57dbbd71 2024-03-22T05:54:49.657 INFO:teuthology.orchestra.run.smithi005.stdout:iscsi.foo.smithi005.mcdwbp smithi005 running (88s) 83s ago 33m - - 3.7 aa461e7c5e7f 8a50f2d11e66 2024-03-22T05:54:49.657 INFO:teuthology.orchestra.run.smithi005.stdout:mgr.x smithi186 *:8443,9283,8765 running (13m) 2m ago 38m 454M - 19.0.0-2299-gea01217f aa461e7c5e7f c556421a73f3 2024-03-22T05:54:49.657 INFO:teuthology.orchestra.run.smithi005.stdout:mgr.y smithi005 *:8443,9283,8765 running (23m) 83s ago 40m - - 19.0.0-2299-gea01217f aa461e7c5e7f 62a813eebc0d 2024-03-22T05:54:49.657 INFO:teuthology.orchestra.run.smithi005.stdout:mon.a smithi005 running (10m) 83s ago 40m - 2048M 19.0.0-2299-gea01217f aa461e7c5e7f 111a2aa24dd9 2024-03-22T05:54:49.657 INFO:teuthology.orchestra.run.smithi005.stdout:mon.b smithi186 running (10m) 2m ago 38m 55.8M 2048M 19.0.0-2299-gea01217f aa461e7c5e7f c1798feac15b 2024-03-22T05:54:49.657 INFO:teuthology.orchestra.run.smithi005.stdout:mon.c smithi005 running (9m) 83s ago 39m - 2048M 19.0.0-2299-gea01217f aa461e7c5e7f 708c72ff74f5 2024-03-22T05:54:49.657 INFO:teuthology.orchestra.run.smithi005.stdout:node-exporter.a smithi005 *:9100 running (13m) 83s ago 34m - - 1.5.0 0da6a335fe13 f635cd72b9d3 2024-03-22T05:54:49.657 INFO:teuthology.orchestra.run.smithi005.stdout:node-exporter.b smithi186 *:9100 running (12m) 2m ago 34m 14.7M - 1.5.0 0da6a335fe13 d7e0b3d20191 2024-03-22T05:54:49.657 INFO:teuthology.orchestra.run.smithi005.stdout:osd.0 smithi005 running (7m) 83s ago 38m - 2993M 19.0.0-2299-gea01217f aa461e7c5e7f 31b6cdee5f5b 2024-03-22T05:54:49.657 INFO:teuthology.orchestra.run.smithi005.stdout:osd.1 smithi005 running (7m) 83s ago 38m - 2993M 19.0.0-2299-gea01217f aa461e7c5e7f 7d21bc9fd3c2 2024-03-22T05:54:49.657 INFO:teuthology.orchestra.run.smithi005.stdout:osd.2 smithi005 running (6m) 83s ago 37m - 2993M 19.0.0-2299-gea01217f aa461e7c5e7f 628204cb7c98 2024-03-22T05:54:49.657 INFO:teuthology.orchestra.run.smithi005.stdout:osd.3 smithi005 running (5m) 83s ago 37m - 2993M 19.0.0-2299-gea01217f aa461e7c5e7f d31a01a0230c 2024-03-22T05:54:49.657 INFO:teuthology.orchestra.run.smithi005.stdout:osd.4 smithi186 running (4m) 2m ago 36m 63.4M 3248M 19.0.0-2299-gea01217f aa461e7c5e7f 8f7619b726f3 2024-03-22T05:54:49.657 INFO:teuthology.orchestra.run.smithi005.stdout:osd.5 smithi186 running (4m) 2m ago 36m 53.8M 3248M 19.0.0-2299-gea01217f aa461e7c5e7f b80f7eceb2d7 2024-03-22T05:54:49.657 INFO:teuthology.orchestra.run.smithi005.stdout:osd.6 smithi186 running (3m) 2m ago 36m 55.5M 3248M 19.0.0-2299-gea01217f aa461e7c5e7f db7c81962c65 2024-03-22T05:54:49.657 INFO:teuthology.orchestra.run.smithi005.stdout:osd.7 smithi186 running (3m) 2m ago 35m 50.0M 3248M 19.0.0-2299-gea01217f aa461e7c5e7f fcb46ba843ae 2024-03-22T05:54:49.658 INFO:teuthology.orchestra.run.smithi005.stdout:prometheus.a smithi186 *:9095 running (12m) 2m ago 35m 72.0M - 2.43.0 a07b618ecd1d a5a043588d6d 2024-03-22T05:54:49.658 INFO:teuthology.orchestra.run.smithi005.stdout:rgw.foo.smithi005.exdhqu smithi005 *:8000 running (2m) 83s ago 33m - - 19.0.0-2299-gea01217f aa461e7c5e7f aa3dec0ece5c 2024-03-22T05:54:49.658 INFO:teuthology.orchestra.run.smithi005.stdout:rgw.foo.smithi186.mcgodk smithi186 *:8000 running (2m) 2m ago 33m 75.9M - 19.0.0-2299-gea01217f aa461e7c5e7f c7967b92514a 2024-03-22T05:54:49.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:54:49 smithi186 ceph-mon[67903]: pgmap v395: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:54:50.337 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph versions' 2024-03-22T05:54:51.633 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:54:51 smithi005 ceph-mon[89904]: from='client.64885 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:54:51.633 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:54:51 smithi005 ceph-mon[89904]: pgmap v396: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:54:51.633 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:54:51 smithi005 ceph-mon[84705]: from='client.64885 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:54:51.633 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:54:51 smithi005 ceph-mon[84705]: pgmap v396: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:54:51.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:54:51 smithi186 ceph-mon[67903]: from='client.64885 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:54:51.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:54:51 smithi186 ceph-mon[67903]: pgmap v396: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:54:52.449 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-03-22T05:54:52.450 INFO:teuthology.orchestra.run.smithi005.stdout: "mon": { 2024-03-22T05:54:52.450 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev)": 3 2024-03-22T05:54:52.450 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:54:52.450 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr": { 2024-03-22T05:54:52.450 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev)": 2 2024-03-22T05:54:52.450 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:54:52.450 INFO:teuthology.orchestra.run.smithi005.stdout: "osd": { 2024-03-22T05:54:52.450 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev)": 8 2024-03-22T05:54:52.450 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:54:52.450 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw": { 2024-03-22T05:54:52.450 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev)": 2 2024-03-22T05:54:52.450 INFO:teuthology.orchestra.run.smithi005.stdout: }, 2024-03-22T05:54:52.450 INFO:teuthology.orchestra.run.smithi005.stdout: "overall": { 2024-03-22T05:54:52.450 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev)": 15 2024-03-22T05:54:52.451 INFO:teuthology.orchestra.run.smithi005.stdout: } 2024-03-22T05:54:52.451 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-03-22T05:54:53.151 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph orch upgrade status' 2024-03-22T05:54:53.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:54:53 smithi005 ceph-mon[84705]: pgmap v397: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:54:53.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:54:53 smithi005 ceph-mon[84705]: from='client.44657 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:54:53.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:54:53 smithi005 ceph-mon[84705]: from='client.? 172.21.15.5:0/3350069031' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:54:53.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:54:53 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:54:53.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:54:53 smithi005 ceph-mon[89904]: pgmap v397: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:54:53.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:54:53 smithi005 ceph-mon[89904]: from='client.44657 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:54:53.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:54:53 smithi005 ceph-mon[89904]: from='client.? 172.21.15.5:0/3350069031' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:54:53.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:54:53 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:54:53.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:54:53 smithi186 ceph-mon[67903]: pgmap v397: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:54:53.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:54:53 smithi186 ceph-mon[67903]: from='client.44657 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:54:53.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:54:53 smithi186 ceph-mon[67903]: from='client.? 172.21.15.5:0/3350069031' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:54:53.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:54:53 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:54:55.300 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-03-22T05:54:55.301 INFO:teuthology.orchestra.run.smithi005.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9", 2024-03-22T05:54:55.301 INFO:teuthology.orchestra.run.smithi005.stdout: "in_progress": true, 2024-03-22T05:54:55.301 INFO:teuthology.orchestra.run.smithi005.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-22T05:54:55.301 INFO:teuthology.orchestra.run.smithi005.stdout: "services_complete": [ 2024-03-22T05:54:55.301 INFO:teuthology.orchestra.run.smithi005.stdout: "iscsi", 2024-03-22T05:54:55.301 INFO:teuthology.orchestra.run.smithi005.stdout: "mon", 2024-03-22T05:54:55.301 INFO:teuthology.orchestra.run.smithi005.stdout: "rgw", 2024-03-22T05:54:55.301 INFO:teuthology.orchestra.run.smithi005.stdout: "osd", 2024-03-22T05:54:55.301 INFO:teuthology.orchestra.run.smithi005.stdout: "mgr" 2024-03-22T05:54:55.301 INFO:teuthology.orchestra.run.smithi005.stdout: ], 2024-03-22T05:54:55.301 INFO:teuthology.orchestra.run.smithi005.stdout: "progress": "16/21 daemons upgraded", 2024-03-22T05:54:55.301 INFO:teuthology.orchestra.run.smithi005.stdout: "message": "Error: UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi005.mcdwbp on host smithi005 failed.", 2024-03-22T05:54:55.302 INFO:teuthology.orchestra.run.smithi005.stdout: "is_paused": true 2024-03-22T05:54:55.302 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-03-22T05:54:55.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:54:55 smithi005 ceph-mon[84705]: pgmap v398: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:54:55.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:54:55 smithi005 ceph-mon[89904]: pgmap v398: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:54:55.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:54:55 smithi186 ceph-mon[67903]: pgmap v398: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:54:55.917 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph health detail' 2024-03-22T05:54:56.563 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:54:56 smithi005 ceph-mon[84705]: from='client.54972 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:54:56.563 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:54:56 smithi005 ceph-mon[89904]: from='client.54972 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:54:56.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:54:56 smithi186 ceph-mon[67903]: from='client.54972 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:54:57.202 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:54:56 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:54:56] "GET /metrics HTTP/1.1" 200 38472 "" "Prometheus/2.43.0" 2024-03-22T05:54:57.522 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:54:57 smithi005 ceph-mon[84705]: pgmap v399: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:54:57.522 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:54:57 smithi005 ceph-mon[89904]: pgmap v399: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:54:57.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:54:57 smithi186 ceph-mon[67903]: pgmap v399: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:54:58.055 INFO:teuthology.orchestra.run.smithi005.stdout:HEALTH_WARN Upgrading daemon iscsi.foo.smithi005.mcdwbp on host smithi005 failed. 2024-03-22T05:54:58.055 INFO:teuthology.orchestra.run.smithi005.stdout:[WRN] UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi005.mcdwbp on host smithi005 failed. 2024-03-22T05:54:58.055 INFO:teuthology.orchestra.run.smithi005.stdout: Upgrade daemon: iscsi.foo.smithi005.mcdwbp: cephadm exited with an error code: 1, stderr: Redeploy daemon iscsi.foo.smithi005.mcdwbp ... 2024-03-22T05:54:58.056 INFO:teuthology.orchestra.run.smithi005.stdout:Creating ceph-iscsi config... 2024-03-22T05:54:58.056 INFO:teuthology.orchestra.run.smithi005.stdout:Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/iscsi-gateway.cfg 2024-03-22T05:54:58.056 INFO:teuthology.orchestra.run.smithi005.stdout:Write file: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/iscsi.foo.smithi005.mcdwbp/tcmu-runner-entrypoint.sh 2024-03-22T05:54:58.056 INFO:teuthology.orchestra.run.smithi005.stdout:Failed to trim old cgroups /sys/fs/cgroup/system.slice/system-ceph\x2df59f36e2\x2de80a\x2d11ee\x2d95cd\x2d87774f69a715.slice/ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service 2024-03-22T05:54:58.056 INFO:teuthology.orchestra.run.smithi005.stdout:Non-zero exit code 1 from systemctl start ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp 2024-03-22T05:54:58.056 INFO:teuthology.orchestra.run.smithi005.stdout:systemctl: stderr Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:54:58.056 INFO:teuthology.orchestra.run.smithi005.stdout:systemctl: stderr See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:54:58.056 INFO:teuthology.orchestra.run.smithi005.stdout:Traceback (most recent call last): 2024-03-22T05:54:58.056 INFO:teuthology.orchestra.run.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-22T05:54:58.056 INFO:teuthology.orchestra.run.smithi005.stdout: return _run_code(code, main_globals, None, 2024-03-22T05:54:58.056 INFO:teuthology.orchestra.run.smithi005.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-22T05:54:58.056 INFO:teuthology.orchestra.run.smithi005.stdout: exec(code, run_globals) 2024-03-22T05:54:58.057 INFO:teuthology.orchestra.run.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5608, in 2024-03-22T05:54:58.057 INFO:teuthology.orchestra.run.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 5596, in main 2024-03-22T05:54:58.057 INFO:teuthology.orchestra.run.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3096, in command_deploy_from 2024-03-22T05:54:58.057 INFO:teuthology.orchestra.run.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3131, in _common_deploy 2024-03-22T05:54:58.057 INFO:teuthology.orchestra.run.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 3151, in _deploy_daemon_container 2024-03-22T05:54:58.057 INFO:teuthology.orchestra.run.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1087, in deploy_daemon 2024-03-22T05:54:58.057 INFO:teuthology.orchestra.run.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/__main__.py", line 1223, in deploy_daemon_units 2024-03-22T05:54:58.057 INFO:teuthology.orchestra.run.smithi005.stdout: File "/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/cephadm.acec822820c266f3b8aa58abfb4b6d0a35a711292410e5a77d4e545630aefcc7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-22T05:54:58.057 INFO:teuthology.orchestra.run.smithi005.stdout:RuntimeError: Failed command: systemctl start ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp: Job for ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service failed because the control process exited with error code. 2024-03-22T05:54:58.057 INFO:teuthology.orchestra.run.smithi005.stdout:See "systemctl status ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" and "journalctl -xeu ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@iscsi.foo.smithi005.mcdwbp.service" for details. 2024-03-22T05:54:58.666 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:54:58 smithi005 ceph-mon[84705]: from='client.? 172.21.15.5:0/3338786638' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T05:54:58.666 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:54:58 smithi005 ceph-mon[89904]: from='client.? 172.21.15.5:0/3338786638' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T05:54:58.691 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 1'"'"'' 2024-03-22T05:54:58.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:54:58 smithi186 ceph-mon[67903]: from='client.? 172.21.15.5:0/3338786638' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-22T05:54:59.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:54:59 smithi005 ceph-mon[84705]: pgmap v400: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:54:59.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:54:59 smithi005 ceph-mon[89904]: pgmap v400: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:54:59.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:54:59 smithi186 ceph-mon[67903]: pgmap v400: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:55:00.905 INFO:teuthology.orchestra.run.smithi005.stdout:true 2024-03-22T05:55:01.550 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:55:01 smithi005 ceph-mon[89904]: pgmap v401: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:55:01.550 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:55:01 smithi005 ceph-mon[89904]: from='client.? 172.21.15.5:0/3145947383' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:55:01.550 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:55:01 smithi005 ceph-mon[84705]: pgmap v401: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:55:01.550 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:55:01 smithi005 ceph-mon[84705]: from='client.? 172.21.15.5:0/3145947383' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:55:01.573 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph versions | jq -e '"'"'.overall | keys'"'"' | grep $sha1' 2024-03-22T05:55:01.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:55:01 smithi186 ceph-mon[67903]: pgmap v401: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:55:01.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:55:01 smithi186 ceph-mon[67903]: from='client.? 172.21.15.5:0/3145947383' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:55:03.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:55:03 smithi005 ceph-mon[84705]: pgmap v402: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:55:03.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:55:03 smithi005 ceph-mon[84705]: from='client.44657 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:55:03.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:55:03 smithi005 ceph-mon[89904]: pgmap v402: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:55:03.453 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:55:03 smithi005 ceph-mon[89904]: from='client.44657 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:55:03.675 INFO:teuthology.orchestra.run.smithi005.stdout: "ceph version 19.0.0-2299-gea01217f (ea01217f8a79678e64ef6a5d68a043a4ef21ca37) squid (dev)" 2024-03-22T05:55:03.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:55:03 smithi186 ceph-mon[67903]: pgmap v402: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:55:03.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:55:03 smithi186 ceph-mon[67903]: from='client.44657 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:55:04.276 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -e sha1=ea01217f8a79678e64ef6a5d68a043a4ef21ca37 -- bash -c 'ceph orch ls | grep '"'"'^osd '"'"'' 2024-03-22T05:55:04.687 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:55:04 smithi005 ceph-mon[84705]: from='client.? 172.21.15.5:0/1168614209' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:55:04.688 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:55:04 smithi005 ceph-mon[89904]: from='client.? 172.21.15.5:0/1168614209' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:55:04.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:55:04 smithi186 ceph-mon[67903]: from='client.? 172.21.15.5:0/1168614209' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-22T05:55:05.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:55:05 smithi005 ceph-mon[84705]: pgmap v403: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:55:05.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:55:05 smithi005 ceph-mon[89904]: pgmap v403: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:55:05.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:55:05 smithi186 ceph-mon[67903]: pgmap v403: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:55:06.492 INFO:teuthology.orchestra.run.smithi005.stdout:osd 8 2m ago - 2024-03-22T05:55:07.074 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-22T05:55:07.085 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi005.front.sepia.ceph.com 2024-03-22T05:55:07.085 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -- bash -c 'ceph orch upgrade ls' 2024-03-22T05:55:07.202 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:55:06 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:55:06] "GET /metrics HTTP/1.1" 200 38472 "" "Prometheus/2.43.0" 2024-03-22T05:55:07.644 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:55:07 smithi005 ceph-mon[84705]: pgmap v404: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:55:07.644 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:55:07 smithi005 ceph-mon[89904]: pgmap v404: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:55:07.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:55:07 smithi186 ceph-mon[67903]: pgmap v404: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:55:08.381 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:55:08 smithi005 ceph-mon[84705]: from='client.54996 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:55:08.381 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:55:08 smithi005 ceph-mon[84705]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:55:08.381 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:55:08 smithi005 ceph-mon[89904]: from='client.54996 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:55:08.382 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:55:08 smithi005 ceph-mon[89904]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:55:08.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:55:08 smithi186 ceph-mon[67903]: from='client.54996 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:55:08.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:55:08 smithi186 ceph-mon[67903]: from='mgr.54108 172.21.15.5:0/2225724236' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-22T05:55:09.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:55:09 smithi005 ceph-mon[84705]: pgmap v405: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:55:09.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:55:09 smithi005 ceph-mon[89904]: pgmap v405: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:55:09.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:55:09 smithi186 ceph-mon[67903]: pgmap v405: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:55:09.898 INFO:teuthology.orchestra.run.smithi005.stdout:{ 2024-03-22T05:55:09.898 INFO:teuthology.orchestra.run.smithi005.stdout: "image": "quay.io/ceph/ceph", 2024-03-22T05:55:09.898 INFO:teuthology.orchestra.run.smithi005.stdout: "registry": "quay.io", 2024-03-22T05:55:09.898 INFO:teuthology.orchestra.run.smithi005.stdout: "bare_image": "ceph/ceph", 2024-03-22T05:55:09.898 INFO:teuthology.orchestra.run.smithi005.stdout: "versions": [] 2024-03-22T05:55:09.899 INFO:teuthology.orchestra.run.smithi005.stdout:} 2024-03-22T05:55:10.596 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:55:10 smithi005 ceph-mon[84705]: from='client.64927 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:55:10.597 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:55:10 smithi005 ceph-mon[89904]: from='client.64927 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:55:10.626 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -- bash -c 'ceph orch upgrade ls --image quay.io/ceph/ceph --show-all-versions | grep 16.2.0' 2024-03-22T05:55:10.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:55:10 smithi186 ceph-mon[67903]: from='client.64927 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:55:11.644 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:55:11 smithi005 ceph-mon[84705]: pgmap v406: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:55:11.645 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:55:11 smithi005 ceph-mon[89904]: pgmap v406: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:55:11.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:55:11 smithi186 ceph-mon[67903]: pgmap v406: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:55:13.565 INFO:teuthology.orchestra.run.smithi005.stdout: "16.2.0", 2024-03-22T05:55:13.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:55:13 smithi005 ceph-mon[84705]: pgmap v407: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:55:13.702 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:55:13 smithi005 ceph-mon[84705]: from='client.44657 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:55:13.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:55:13 smithi005 ceph-mon[89904]: pgmap v407: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:55:13.703 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:55:13 smithi005 ceph-mon[89904]: from='client.44657 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:55:13.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:55:13 smithi186 ceph-mon[67903]: pgmap v407: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-22T05:55:13.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:55:13 smithi186 ceph-mon[67903]: from='client.44657 -' entity='client.iscsi.foo.smithi005.mcdwbp' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-22T05:55:14.242 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 -- bash -c 'ceph orch upgrade ls --image quay.io/ceph/ceph --tags | grep v16.2.2' 2024-03-22T05:55:14.563 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:55:14 smithi005 ceph-mon[89904]: from='client.64933 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:55:14.563 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:55:14 smithi005 ceph-mon[84705]: from='client.64933 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:55:14.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:55:14 smithi186 ceph-mon[67903]: from='client.64933 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-22T05:55:15.644 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:55:15 smithi005 ceph-mon[84705]: pgmap v408: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:55:15.644 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:55:15 smithi005 ceph-mon[89904]: pgmap v408: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:55:15.794 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:55:15 smithi186 ceph-mon[67903]: pgmap v408: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:55:17.007 INFO:teuthology.orchestra.run.smithi005.stdout: "v16.2.2", 2024-03-22T05:55:17.007 INFO:teuthology.orchestra.run.smithi005.stdout: "v16.2.2-20210505", 2024-03-22T05:55:17.334 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:55:16 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: ::ffff:172.21.15.186 - - [22/Mar/2024:05:55:16] "GET /metrics HTTP/1.1" 200 38474 "" "Prometheus/2.43.0" 2024-03-22T05:55:17.659 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:55:17 smithi005 ceph-mon[84705]: pgmap v409: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:55:17.659 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:55:17 smithi005 ceph-mon[89904]: pgmap v409: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:55:17.684 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-22T05:55:17.694 INFO:tasks.cephadm:Teardown begin 2024-03-22T05:55:17.694 DEBUG:teuthology.orchestra.run.smithi005:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-22T05:55:17.725 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-22T05:55:17.757 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:55:17 smithi186 ceph-mon[67903]: pgmap v409: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-22T05:55:17.762 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-22T05:55:17.762 DEBUG:teuthology.orchestra.run.smithi005:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-22T05:55:17.780 DEBUG:teuthology.orchestra.run.smithi186:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-22T05:55:17.819 INFO:tasks.cephadm:Stopping all daemons... 2024-03-22T05:55:17.819 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-22T05:55:17.820 DEBUG:teuthology.orchestra.run.smithi005:> sudo systemctl stop ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@mon.a 2024-03-22T05:55:17.952 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:55:17 smithi005 systemd[1]: Stopping Ceph mon.a for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:55:18.452 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:55:18 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mon-a[84700]: 2024-03-22T05:55:18.112+0000 7f83fbf60700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T05:55:18.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:55:18 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mon-a[84700]: 2024-03-22T05:55:18.112+0000 7f83fbf60700 -1 mon.a@0(leader) e4 *** Got Signal Terminated *** 2024-03-22T05:55:18.453 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:55:18 smithi005 podman[136703]: 2024-03-22 05:55:18.184917831 +0000 UTC m=+0.178123353 container died 111a2aa24dd942b914010b3779e02df03411258da7eb5f9e8c8139f5ca03e9ad (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mon-a, CEPH_POINT_RELEASE=, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-22T05:55:18.839 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:55:18 smithi005 podman[136703]: 2024-03-22 05:55:18.554852872 +0000 UTC m=+0.548058401 container cleanup 111a2aa24dd942b914010b3779e02df03411258da7eb5f9e8c8139f5ca03e9ad (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mon-a, org.label-schema.build-date=20240319, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-22T05:55:18.839 INFO:journalctl@ceph.mon.a.smithi005.stdout:Mar 22 05:55:18 smithi005 bash[136703]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mon-a 2024-03-22T05:55:18.976 DEBUG:teuthology.orchestra.run.smithi005:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@mon.a.service' 2024-03-22T05:55:19.027 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-22T05:55:19.027 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-22T05:55:19.027 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-03-22T05:55:19.027 DEBUG:teuthology.orchestra.run.smithi005:> sudo systemctl stop ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@mon.c 2024-03-22T05:55:19.202 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:55:19 smithi005 systemd[1]: Stopping Ceph mon.c for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:55:19.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:55:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mon-c[89900]: 2024-03-22T05:55:19.325+0000 7fb93c1d0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T05:55:19.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:55:19 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mon-c[89900]: 2024-03-22T05:55:19.325+0000 7fb93c1d0700 -1 mon.c@1(peon) e4 *** Got Signal Terminated *** 2024-03-22T05:55:19.702 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:55:19 smithi005 podman[136824]: 2024-03-22 05:55:19.550834988 +0000 UTC m=+0.340681843 container died 708c72ff74f5b841a258f70d64a4f638b6da5c5ea37a4ba779c0151f63a841c5 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mon-c, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319) 2024-03-22T05:55:20.193 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:55:19 smithi005 podman[136824]: 2024-03-22 05:55:19.900898025 +0000 UTC m=+0.690744882 container cleanup 708c72ff74f5b841a258f70d64a4f638b6da5c5ea37a4ba779c0151f63a841c5 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mon-c, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-22T05:55:20.193 INFO:journalctl@ceph.mon.c.smithi005.stdout:Mar 22 05:55:19 smithi005 bash[136824]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mon-c 2024-03-22T05:55:20.320 DEBUG:teuthology.orchestra.run.smithi005:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@mon.c.service' 2024-03-22T05:55:20.356 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-22T05:55:20.356 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-03-22T05:55:20.357 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-03-22T05:55:20.357 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@mon.b 2024-03-22T05:55:20.637 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:55:20 smithi186 systemd[1]: Stopping Ceph mon.b for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:55:20.920 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:55:20 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mon-b[67899]: 2024-03-22T05:55:20.635+0000 7fdd80a0d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T05:55:20.920 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:55:20 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mon-b[67899]: 2024-03-22T05:55:20.635+0000 7fdd80a0d700 -1 mon.b@2(peon) e4 *** Got Signal Terminated *** 2024-03-22T05:55:20.920 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:55:20 smithi186 podman[101122]: 2024-03-22 05:55:20.6979277 +0000 UTC m=+0.146473502 container died c1798feac15b559b751d8adbbc6ff346ae094c5e47fdd03cbfa15609427fac0b (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mon-b, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS) 2024-03-22T05:55:21.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:55:21 smithi186 podman[101122]: 2024-03-22 05:55:21.094819659 +0000 UTC m=+0.543365461 container cleanup c1798feac15b559b751d8adbbc6ff346ae094c5e47fdd03cbfa15609427fac0b (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mon-b, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-22T05:55:21.294 INFO:journalctl@ceph.mon.b.smithi186.stdout:Mar 22 05:55:21 smithi186 bash[101122]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mon-b 2024-03-22T05:55:21.508 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@mon.b.service' 2024-03-22T05:55:21.556 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-22T05:55:21.556 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-03-22T05:55:21.556 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-22T05:55:21.556 DEBUG:teuthology.orchestra.run.smithi005:> sudo systemctl stop ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@mgr.y 2024-03-22T05:55:21.880 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:55:21 smithi005 systemd[1]: Stopping Ceph mgr.y for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:55:22.189 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:55:21 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:55:21.879+0000 7f7f7cdda700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T05:55:22.189 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:55:21 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y[63229]: 2024-03-22T05:55:21.879+0000 7f7f7cdda700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-22T05:55:22.189 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:55:21 smithi005 podman[136944]: 2024-03-22 05:55:21.911265562 +0000 UTC m=+0.166405838 container died 62a813eebc0db2b81d3553d8158695697ee8f9dd162b89616f3e4e05a385cffb (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, ceph=True) 2024-03-22T05:55:22.452 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:55:22 smithi005 podman[136944]: 2024-03-22 05:55:22.263850482 +0000 UTC m=+0.518990746 container cleanup 62a813eebc0db2b81d3553d8158695697ee8f9dd162b89616f3e4e05a385cffb (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-22T05:55:22.452 INFO:journalctl@ceph.mgr.y.smithi005.stdout:Mar 22 05:55:22 smithi005 bash[136944]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-y 2024-03-22T05:55:22.684 DEBUG:teuthology.orchestra.run.smithi005:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@mgr.y.service' 2024-03-22T05:55:22.721 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-22T05:55:22.721 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-22T05:55:22.721 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-22T05:55:22.721 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@mgr.x 2024-03-22T05:55:23.038 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:55:22 smithi186 systemd[1]: Stopping Ceph mgr.x for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:55:23.294 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:55:23 smithi186 podman[101243]: 2024-03-22 05:55:23.038983916 +0000 UTC m=+0.124580905 container died c556421a73f3781265649ad0e2056caac1cc5811fe40cfbecef68766d755c7b3 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=) 2024-03-22T05:55:23.660 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:55:23 smithi186 podman[101243]: 2024-03-22 05:55:23.483664621 +0000 UTC m=+0.569261607 container cleanup c556421a73f3781265649ad0e2056caac1cc5811fe40cfbecef68766d755c7b3 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0) 2024-03-22T05:55:23.660 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:55:23 smithi186 bash[101243]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x 2024-03-22T05:55:23.901 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@mgr.x.service' 2024-03-22T05:55:23.926 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:55:23 smithi186 podman[101256]: 2024-03-22 05:55:23.660240306 +0000 UTC m=+0.620056812 container remove c556421a73f3781265649ad0e2056caac1cc5811fe40cfbecef68766d755c7b3 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-mgr-x, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-22T05:55:23.926 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:55:23 smithi186 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-03-22T05:55:23.926 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:55:23 smithi186 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@mgr.x.service: Failed with result 'exit-code'. 2024-03-22T05:55:23.926 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:55:23 smithi186 systemd[1]: Stopped Ceph mgr.x for f59f36e2-e80a-11ee-95cd-87774f69a715. 2024-03-22T05:55:23.927 INFO:journalctl@ceph.mgr.x.smithi186.stdout:Mar 22 05:55:23 smithi186 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@mgr.x.service: Consumed 23.374s CPU time. 2024-03-22T05:55:23.939 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-22T05:55:23.939 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-22T05:55:23.939 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-22T05:55:23.939 DEBUG:teuthology.orchestra.run.smithi005:> sudo systemctl stop ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@osd.0 2024-03-22T05:55:24.226 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 22 05:55:23 smithi005 systemd[1]: Stopping Ceph osd.0 for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:55:24.702 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 22 05:55:24 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-0[97978]: 2024-03-22T05:55:24.225+0000 7f8322d74700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T05:55:24.702 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 22 05:55:24 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-0[97978]: 2024-03-22T05:55:24.225+0000 7f8322d74700 -1 osd.0 158 *** Got signal Terminated *** 2024-03-22T05:55:24.703 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 22 05:55:24 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-0[97978]: 2024-03-22T05:55:24.225+0000 7f8322d74700 -1 osd.0 158 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-22T05:55:29.994 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 22 05:55:29 smithi005 podman[137064]: 2024-03-22 05:55:29.715477899 +0000 UTC m=+5.582913624 container died 31b6cdee5f5b9f92efc9cf537c9ee43b9bdbbc3345f06de68023d7618d3b402c (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-0, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-22T05:55:30.344 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 22 05:55:30 smithi005 podman[137064]: 2024-03-22 05:55:30.093061945 +0000 UTC m=+5.960497669 container cleanup 31b6cdee5f5b9f92efc9cf537c9ee43b9bdbbc3345f06de68023d7618d3b402c (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True) 2024-03-22T05:55:30.344 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 22 05:55:30 smithi005 bash[137064]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-0 2024-03-22T05:55:30.644 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 22 05:55:30 smithi005 podman[137082]: 2024-03-22 05:55:30.344348955 +0000 UTC m=+0.626627511 container remove 31b6cdee5f5b9f92efc9cf537c9ee43b9bdbbc3345f06de68023d7618d3b402c (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-0, org.label-schema.build-date=20240319, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-03-22T05:55:30.939 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 22 05:55:30 smithi005 podman[137173]: 2024-03-22 05:55:30.548146413 +0000 UTC m=+0.017507695 image pull aa461e7c5e7f322b169c789b389d594b994ea7ffcbc9e24e563a3ec9b69bc3c1 quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9 2024-03-22T05:55:30.939 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 22 05:55:30 smithi005 podman[137173]: 2024-03-22 05:55:30.653423304 +0000 UTC m=+0.122784588 container create a9808d9b3fdcde446899d29a466696c618e991e0fe8db9c6779fd1a1f508dc46 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-0-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217) 2024-03-22T05:55:30.939 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 22 05:55:30 smithi005 podman[137173]: 2024-03-22 05:55:30.864698669 +0000 UTC m=+0.334059954 container init a9808d9b3fdcde446899d29a466696c618e991e0fe8db9c6779fd1a1f508dc46 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-0-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, org.label-schema.schema-version=1.0) 2024-03-22T05:55:30.939 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 22 05:55:30 smithi005 podman[137173]: 2024-03-22 05:55:30.867729419 +0000 UTC m=+0.337090714 container start a9808d9b3fdcde446899d29a466696c618e991e0fe8db9c6779fd1a1f508dc46 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-0-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-22T05:55:30.939 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 22 05:55:30 smithi005 podman[137173]: 2024-03-22 05:55:30.93950729 +0000 UTC m=+0.408868588 container attach a9808d9b3fdcde446899d29a466696c618e991e0fe8db9c6779fd1a1f508dc46 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-0-deactivate, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-22T05:55:31.191 INFO:journalctl@ceph.osd.0.smithi005.stdout:Mar 22 05:55:31 smithi005 podman[137173]: 2024-03-22 05:55:31.16767504 +0000 UTC m=+0.637036313 container died a9808d9b3fdcde446899d29a466696c618e991e0fe8db9c6779fd1a1f508dc46 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-0-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, GIT_CLEAN=True, org.label-schema.build-date=20240319) 2024-03-22T05:55:31.793 DEBUG:teuthology.orchestra.run.smithi005:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@osd.0.service' 2024-03-22T05:55:31.835 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-22T05:55:31.835 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-22T05:55:31.835 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-22T05:55:31.835 DEBUG:teuthology.orchestra.run.smithi005:> sudo systemctl stop ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@osd.1 2024-03-22T05:55:32.197 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 22 05:55:31 smithi005 systemd[1]: Stopping Ceph osd.1 for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:55:32.452 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 22 05:55:32 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-1[103344]: 2024-03-22T05:55:32.196+0000 7f67a7bf1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T05:55:32.452 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 22 05:55:32 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-1[103344]: 2024-03-22T05:55:32.196+0000 7f67a7bf1700 -1 osd.1 158 *** Got signal Terminated *** 2024-03-22T05:55:32.452 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 22 05:55:32 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-1[103344]: 2024-03-22T05:55:32.196+0000 7f67a7bf1700 -1 osd.1 158 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-22T05:55:37.952 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 22 05:55:37 smithi005 podman[137289]: 2024-03-22 05:55:37.675249164 +0000 UTC m=+5.579763538 container died 7d21bc9fd3c2343f595dd18fa339b995382d09d3a2464b6173f250dde69c98fa (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-1, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS) 2024-03-22T05:55:38.452 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 22 05:55:37 smithi005 podman[137289]: 2024-03-22 05:55:37.995808862 +0000 UTC m=+5.900323236 container cleanup 7d21bc9fd3c2343f595dd18fa339b995382d09d3a2464b6173f250dde69c98fa (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-1, maintainer=Guillaume Abrioux , RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True) 2024-03-22T05:55:38.452 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 22 05:55:37 smithi005 bash[137289]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-1 2024-03-22T05:55:38.452 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 22 05:55:38 smithi005 podman[137306]: 2024-03-22 05:55:38.213753329 +0000 UTC m=+0.532589292 container remove 7d21bc9fd3c2343f595dd18fa339b995382d09d3a2464b6173f250dde69c98fa (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-1, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5) 2024-03-22T05:55:38.925 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 22 05:55:38 smithi005 podman[137398]: 2024-03-22 05:55:38.520344519 +0000 UTC m=+0.019123824 image pull aa461e7c5e7f322b169c789b389d594b994ea7ffcbc9e24e563a3ec9b69bc3c1 quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9 2024-03-22T05:55:38.925 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 22 05:55:38 smithi005 podman[137398]: 2024-03-22 05:55:38.623034266 +0000 UTC m=+0.121813583 container create ebc1f18f5af98319606e38e82e0bf63f42de2b42dcb8befebf8c04e3ff177d9f (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-1-deactivate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, org.label-schema.vendor=CentOS) 2024-03-22T05:55:38.926 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 22 05:55:38 smithi005 podman[137398]: 2024-03-22 05:55:38.841974237 +0000 UTC m=+0.340753525 container init ebc1f18f5af98319606e38e82e0bf63f42de2b42dcb8befebf8c04e3ff177d9f (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-22T05:55:38.926 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 22 05:55:38 smithi005 podman[137398]: 2024-03-22 05:55:38.845045434 +0000 UTC m=+0.343824726 container start ebc1f18f5af98319606e38e82e0bf63f42de2b42dcb8befebf8c04e3ff177d9f (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-1-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-22T05:55:39.202 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 22 05:55:38 smithi005 podman[137398]: 2024-03-22 05:55:38.92527507 +0000 UTC m=+0.424054361 container attach ebc1f18f5af98319606e38e82e0bf63f42de2b42dcb8befebf8c04e3ff177d9f (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-22T05:55:39.203 INFO:journalctl@ceph.osd.1.smithi005.stdout:Mar 22 05:55:39 smithi005 podman[137398]: 2024-03-22 05:55:39.139787152 +0000 UTC m=+0.638566428 container died ebc1f18f5af98319606e38e82e0bf63f42de2b42dcb8befebf8c04e3ff177d9f (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-1-deactivate, org.label-schema.build-date=20240319, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, io.buildah.version=1.33.5) 2024-03-22T05:55:39.705 DEBUG:teuthology.orchestra.run.smithi005:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@osd.1.service' 2024-03-22T05:55:39.750 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-22T05:55:39.750 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-22T05:55:39.750 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-22T05:55:39.751 DEBUG:teuthology.orchestra.run.smithi005:> sudo systemctl stop ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@osd.2 2024-03-22T05:55:39.952 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 22 05:55:39 smithi005 systemd[1]: Stopping Ceph osd.2 for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:55:40.452 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 22 05:55:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-2[110164]: 2024-03-22T05:55:40.056+0000 7f063cce5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T05:55:40.452 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 22 05:55:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-2[110164]: 2024-03-22T05:55:40.056+0000 7f063cce5700 -1 osd.2 158 *** Got signal Terminated *** 2024-03-22T05:55:40.452 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 22 05:55:40 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-2[110164]: 2024-03-22T05:55:40.056+0000 7f063cce5700 -1 osd.2 158 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-22T05:55:45.648 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 22 05:55:45 smithi005 podman[137502]: 2024-03-22 05:55:45.376130768 +0000 UTC m=+5.407373272 container died 628204cb7c98642d56569171045efcb658293c1daaaf2448e9d6e12f142d9dda (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-22T05:55:45.952 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 22 05:55:45 smithi005 podman[137502]: 2024-03-22 05:55:45.831461714 +0000 UTC m=+5.862704267 container cleanup 628204cb7c98642d56569171045efcb658293c1daaaf2448e9d6e12f142d9dda (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-22T05:55:45.952 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 22 05:55:45 smithi005 bash[137502]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-2 2024-03-22T05:55:46.409 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 22 05:55:46 smithi005 podman[137519]: 2024-03-22 05:55:46.099654472 +0000 UTC m=+0.725982128 container remove 628204cb7c98642d56569171045efcb658293c1daaaf2448e9d6e12f142d9dda (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-22T05:55:46.694 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 22 05:55:46 smithi005 podman[137613]: 2024-03-22 05:55:46.312286386 +0000 UTC m=+0.017554388 image pull aa461e7c5e7f322b169c789b389d594b994ea7ffcbc9e24e563a3ec9b69bc3c1 quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9 2024-03-22T05:55:46.694 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 22 05:55:46 smithi005 podman[137613]: 2024-03-22 05:55:46.416921309 +0000 UTC m=+0.122189283 container create b29a959969f7dc8487f2fe42ae44571e153f6da74636f63fdfb6713703aea393 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-2-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-22T05:55:46.694 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 22 05:55:46 smithi005 podman[137613]: 2024-03-22 05:55:46.612617595 +0000 UTC m=+0.317885588 container init b29a959969f7dc8487f2fe42ae44571e153f6da74636f63fdfb6713703aea393 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-2-deactivate, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-03-22T05:55:46.694 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 22 05:55:46 smithi005 podman[137613]: 2024-03-22 05:55:46.616125105 +0000 UTC m=+0.321393080 container start b29a959969f7dc8487f2fe42ae44571e153f6da74636f63fdfb6713703aea393 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-2-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0) 2024-03-22T05:55:46.694 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 22 05:55:46 smithi005 podman[137613]: 2024-03-22 05:55:46.694360386 +0000 UTC m=+0.399628374 container attach b29a959969f7dc8487f2fe42ae44571e153f6da74636f63fdfb6713703aea393 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-2-deactivate, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-22T05:55:46.952 INFO:journalctl@ceph.osd.2.smithi005.stdout:Mar 22 05:55:46 smithi005 podman[137613]: 2024-03-22 05:55:46.917784407 +0000 UTC m=+0.623052384 container died b29a959969f7dc8487f2fe42ae44571e153f6da74636f63fdfb6713703aea393 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-2-deactivate, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-22T05:55:47.521 DEBUG:teuthology.orchestra.run.smithi005:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@osd.2.service' 2024-03-22T05:55:47.563 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-22T05:55:47.564 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-22T05:55:47.564 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-22T05:55:47.564 DEBUG:teuthology.orchestra.run.smithi005:> sudo systemctl stop ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@osd.3 2024-03-22T05:55:47.761 INFO:journalctl@ceph.osd.3.smithi005.stdout:Mar 22 05:55:47 smithi005 systemd[1]: Stopping Ceph osd.3 for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:55:48.202 INFO:journalctl@ceph.osd.3.smithi005.stdout:Mar 22 05:55:47 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-3[117287]: 2024-03-22T05:55:47.917+0000 7f1e9993a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T05:55:48.202 INFO:journalctl@ceph.osd.3.smithi005.stdout:Mar 22 05:55:47 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-3[117287]: 2024-03-22T05:55:47.917+0000 7f1e9993a700 -1 osd.3 158 *** Got signal Terminated *** 2024-03-22T05:55:48.202 INFO:journalctl@ceph.osd.3.smithi005.stdout:Mar 22 05:55:47 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-3[117287]: 2024-03-22T05:55:47.917+0000 7f1e9993a700 -1 osd.3 158 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-22T05:55:51.044 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:55:50 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6[87212]: 2024-03-22T05:55:50.737+0000 7f1410de7700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:29.197283+0000 front 2024-03-22T05:55:29.197317+0000 (oldest deadline 2024-03-22T05:55:50.297145+0000) 2024-03-22T05:55:52.044 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:55:51 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6[87212]: 2024-03-22T05:55:51.762+0000 7f1410de7700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:29.197283+0000 front 2024-03-22T05:55:29.197317+0000 (oldest deadline 2024-03-22T05:55:50.297145+0000) 2024-03-22T05:55:52.044 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:55:51 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:55:51.704+0000 7f53af753700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:25.432637+0000 front 2024-03-22T05:55:25.432503+0000 (oldest deadline 2024-03-22T05:55:50.732484+0000) 2024-03-22T05:55:52.668 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 22 05:55:52 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-4[78076]: 2024-03-22T05:55:52.356+0000 7f91c535a700 -1 osd.4 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:27.039500+0000 front 2024-03-22T05:55:27.039424+0000 (oldest deadline 2024-03-22T05:55:51.739358+0000) 2024-03-22T05:55:53.044 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:55:52 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6[87212]: 2024-03-22T05:55:52.729+0000 7f1410de7700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:29.197283+0000 front 2024-03-22T05:55:29.197317+0000 (oldest deadline 2024-03-22T05:55:50.297145+0000) 2024-03-22T05:55:53.045 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:55:52 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:55:52.667+0000 7f53af753700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:25.432637+0000 front 2024-03-22T05:55:25.432503+0000 (oldest deadline 2024-03-22T05:55:50.732484+0000) 2024-03-22T05:55:53.202 INFO:journalctl@ceph.osd.3.smithi005.stdout:Mar 22 05:55:52 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-3[117287]: 2024-03-22T05:55:52.819+0000 7f1e91f26700 -1 osd.3 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:26.749529+0000 front 2024-03-22T05:55:26.749564+0000 (oldest deadline 2024-03-22T05:55:52.049718+0000) 2024-03-22T05:55:53.654 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 22 05:55:53 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-4[78076]: 2024-03-22T05:55:53.344+0000 7f91c535a700 -1 osd.4 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:27.039500+0000 front 2024-03-22T05:55:27.039424+0000 (oldest deadline 2024-03-22T05:55:51.739358+0000) 2024-03-22T05:55:53.748 INFO:journalctl@ceph.osd.3.smithi005.stdout:Mar 22 05:55:53 smithi005 podman[137718]: 2024-03-22 05:55:53.466825771 +0000 UTC m=+5.675108756 container died d31a01a0230c2a13213540f3096e04da54ea6bae5316cce09f70ab08e767dbc0 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-3, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-22T05:55:53.748 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:55:53 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[78315]: ts=2024-03-22T05:55:53.554Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi005.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.5:8443: connect: connection refused" 2024-03-22T05:55:53.748 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:55:53 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[78315]: ts=2024-03-22T05:55:53.555Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi186.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.186:8443: connect: connection refused" 2024-03-22T05:55:54.044 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:55:53 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6[87212]: 2024-03-22T05:55:53.729+0000 7f1410de7700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:29.197283+0000 front 2024-03-22T05:55:29.197317+0000 (oldest deadline 2024-03-22T05:55:50.297145+0000) 2024-03-22T05:55:54.044 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:55:53 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:55:53.653+0000 7f53af753700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:25.432637+0000 front 2024-03-22T05:55:25.432503+0000 (oldest deadline 2024-03-22T05:55:50.732484+0000) 2024-03-22T05:55:54.092 INFO:journalctl@ceph.osd.3.smithi005.stdout:Mar 22 05:55:53 smithi005 podman[137718]: 2024-03-22 05:55:53.850591638 +0000 UTC m=+6.058874627 container cleanup d31a01a0230c2a13213540f3096e04da54ea6bae5316cce09f70ab08e767dbc0 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-3, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-22T05:55:54.093 INFO:journalctl@ceph.osd.3.smithi005.stdout:Mar 22 05:55:53 smithi005 bash[137718]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-3 2024-03-22T05:55:54.418 INFO:journalctl@ceph.osd.3.smithi005.stdout:Mar 22 05:55:54 smithi005 podman[137746]: 2024-03-22 05:55:54.09327762 +0000 UTC m=+0.627486040 container remove d31a01a0230c2a13213540f3096e04da54ea6bae5316cce09f70ab08e767dbc0 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-3, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-22T05:55:54.696 INFO:journalctl@ceph.osd.3.smithi005.stdout:Mar 22 05:55:54 smithi005 podman[137842]: 2024-03-22 05:55:54.419102515 +0000 UTC m=+0.107968824 container create 45a9ca8eb8dea1b25772c8f392c225478aed6d34a91d3f138c821fdb55a2b78a (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-3-deactivate, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD) 2024-03-22T05:55:54.696 INFO:journalctl@ceph.osd.3.smithi005.stdout:Mar 22 05:55:54 smithi005 podman[137842]: 2024-03-22 05:55:54.329338923 +0000 UTC m=+0.018205237 image pull aa461e7c5e7f322b169c789b389d594b994ea7ffcbc9e24e563a3ec9b69bc3c1 quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9 2024-03-22T05:55:54.697 INFO:journalctl@ceph.osd.3.smithi005.stdout:Mar 22 05:55:54 smithi005 podman[137842]: 2024-03-22 05:55:54.613654925 +0000 UTC m=+0.302521244 container init 45a9ca8eb8dea1b25772c8f392c225478aed6d34a91d3f138c821fdb55a2b78a (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-3-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-22T05:55:54.697 INFO:journalctl@ceph.osd.3.smithi005.stdout:Mar 22 05:55:54 smithi005 podman[137842]: 2024-03-22 05:55:54.616670475 +0000 UTC m=+0.305536790 container start 45a9ca8eb8dea1b25772c8f392c225478aed6d34a91d3f138c821fdb55a2b78a (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-3-deactivate, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-22T05:55:54.697 INFO:journalctl@ceph.osd.3.smithi005.stdout:Mar 22 05:55:54 smithi005 podman[137842]: 2024-03-22 05:55:54.696728109 +0000 UTC m=+0.385594427 container attach 45a9ca8eb8dea1b25772c8f392c225478aed6d34a91d3f138c821fdb55a2b78a (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-3-deactivate, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, GIT_CLEAN=True) 2024-03-22T05:55:54.702 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 22 05:55:54 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-4[78076]: 2024-03-22T05:55:54.333+0000 7f91c535a700 -1 osd.4 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:27.039500+0000 front 2024-03-22T05:55:27.039424+0000 (oldest deadline 2024-03-22T05:55:51.739358+0000) 2024-03-22T05:55:54.952 INFO:journalctl@ceph.osd.3.smithi005.stdout:Mar 22 05:55:54 smithi005 podman[137842]: 2024-03-22 05:55:54.913933427 +0000 UTC m=+0.602799737 container died 45a9ca8eb8dea1b25772c8f392c225478aed6d34a91d3f138c821fdb55a2b78a (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-3-deactivate, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, GIT_BRANCH=HEAD) 2024-03-22T05:55:55.044 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:55:54 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6[87212]: 2024-03-22T05:55:54.715+0000 7f1410de7700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:29.197283+0000 front 2024-03-22T05:55:29.197317+0000 (oldest deadline 2024-03-22T05:55:50.297145+0000) 2024-03-22T05:55:55.044 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:55:54 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-5[82660]: 2024-03-22T05:55:54.807+0000 7fad9b6cb700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:28.808216+0000 front 2024-03-22T05:55:28.808383+0000 (oldest deadline 2024-03-22T05:55:54.108160+0000) 2024-03-22T05:55:55.044 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:55:54 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:55:54.701+0000 7f53af753700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:25.432637+0000 front 2024-03-22T05:55:25.432503+0000 (oldest deadline 2024-03-22T05:55:50.732484+0000) 2024-03-22T05:55:55.487 DEBUG:teuthology.orchestra.run.smithi005:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@osd.3.service' 2024-03-22T05:55:55.530 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-22T05:55:55.530 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-22T05:55:55.530 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-22T05:55:55.530 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@osd.4 2024-03-22T05:55:55.561 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 22 05:55:55 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-4[78076]: 2024-03-22T05:55:55.306+0000 7f91c535a700 -1 osd.4 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:27.039500+0000 front 2024-03-22T05:55:27.039424+0000 (oldest deadline 2024-03-22T05:55:51.739358+0000) 2024-03-22T05:55:55.830 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 22 05:55:55 smithi186 systemd[1]: Stopping Ceph osd.4 for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:55:55.830 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:55:55 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6[87212]: 2024-03-22T05:55:55.716+0000 7f1410de7700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:29.197283+0000 front 2024-03-22T05:55:29.197317+0000 (oldest deadline 2024-03-22T05:55:50.297145+0000) 2024-03-22T05:55:55.830 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:55:55 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:55:55.716+0000 7f53af753700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:25.432637+0000 front 2024-03-22T05:55:25.432503+0000 (oldest deadline 2024-03-22T05:55:50.732484+0000) 2024-03-22T05:55:56.294 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:55:55 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-5[82660]: 2024-03-22T05:55:55.829+0000 7fad9b6cb700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:28.808216+0000 front 2024-03-22T05:55:28.808383+0000 (oldest deadline 2024-03-22T05:55:54.108160+0000) 2024-03-22T05:55:56.294 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 22 05:55:55 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-4[78076]: 2024-03-22T05:55:55.860+0000 7f91ccd6e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T05:55:56.294 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 22 05:55:55 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-4[78076]: 2024-03-22T05:55:55.860+0000 7f91ccd6e700 -1 osd.4 158 *** Got signal Terminated *** 2024-03-22T05:55:56.294 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 22 05:55:55 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-4[78076]: 2024-03-22T05:55:55.860+0000 7f91ccd6e700 -1 osd.4 158 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-22T05:55:56.689 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 22 05:55:56 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-4[78076]: 2024-03-22T05:55:56.344+0000 7f91c535a700 -1 osd.4 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:27.039500+0000 front 2024-03-22T05:55:27.039424+0000 (oldest deadline 2024-03-22T05:55:51.739358+0000) 2024-03-22T05:55:57.044 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:55:56 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-5[82660]: 2024-03-22T05:55:56.783+0000 7fad9b6cb700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:28.808216+0000 front 2024-03-22T05:55:28.808383+0000 (oldest deadline 2024-03-22T05:55:54.108160+0000) 2024-03-22T05:55:57.044 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:55:56 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6[87212]: 2024-03-22T05:55:56.727+0000 7f1410de7700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:29.197283+0000 front 2024-03-22T05:55:29.197317+0000 (oldest deadline 2024-03-22T05:55:50.297145+0000) 2024-03-22T05:55:57.044 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:55:56 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:55:56.689+0000 7f53af753700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:25.432637+0000 front 2024-03-22T05:55:25.432503+0000 (oldest deadline 2024-03-22T05:55:50.732484+0000) 2024-03-22T05:55:57.696 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 22 05:55:57 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-4[78076]: 2024-03-22T05:55:57.315+0000 7f91c535a700 -1 osd.4 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:27.039500+0000 front 2024-03-22T05:55:27.039424+0000 (oldest deadline 2024-03-22T05:55:51.739358+0000) 2024-03-22T05:55:58.044 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:55:57 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:55:57.696+0000 7f53af753700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:25.432637+0000 front 2024-03-22T05:55:25.432503+0000 (oldest deadline 2024-03-22T05:55:50.732484+0000) 2024-03-22T05:55:58.044 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:55:57 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-5[82660]: 2024-03-22T05:55:57.747+0000 7fad9b6cb700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:28.808216+0000 front 2024-03-22T05:55:28.808383+0000 (oldest deadline 2024-03-22T05:55:54.108160+0000) 2024-03-22T05:55:58.044 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:55:57 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6[87212]: 2024-03-22T05:55:57.730+0000 7f1410de7700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:29.197283+0000 front 2024-03-22T05:55:29.197317+0000 (oldest deadline 2024-03-22T05:55:50.297145+0000) 2024-03-22T05:55:58.689 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 22 05:55:58 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-4[78076]: 2024-03-22T05:55:58.341+0000 7f91c535a700 -1 osd.4 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:27.039500+0000 front 2024-03-22T05:55:27.039424+0000 (oldest deadline 2024-03-22T05:55:51.739358+0000) 2024-03-22T05:55:59.044 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:55:58 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-5[82660]: 2024-03-22T05:55:58.741+0000 7fad9b6cb700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:28.808216+0000 front 2024-03-22T05:55:28.808383+0000 (oldest deadline 2024-03-22T05:55:54.108160+0000) 2024-03-22T05:55:59.044 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:55:58 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6[87212]: 2024-03-22T05:55:58.764+0000 7f1410de7700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:29.197283+0000 front 2024-03-22T05:55:29.197317+0000 (oldest deadline 2024-03-22T05:55:50.297145+0000) 2024-03-22T05:55:59.044 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:55:58 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:55:58.689+0000 7f53af753700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:25.432637+0000 front 2024-03-22T05:55:25.432503+0000 (oldest deadline 2024-03-22T05:55:50.732484+0000) 2024-03-22T05:55:59.642 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 22 05:55:59 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-4[78076]: 2024-03-22T05:55:59.385+0000 7f91c535a700 -1 osd.4 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:27.039500+0000 front 2024-03-22T05:55:27.039424+0000 (oldest deadline 2024-03-22T05:55:51.739358+0000) 2024-03-22T05:55:59.642 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 22 05:55:59 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-4[78076]: 2024-03-22T05:55:59.385+0000 7f91c535a700 -1 osd.4 158 heartbeat_check: no reply from 172.21.15.5:6814 osd.1 since back 2024-03-22T05:55:36.440472+0000 front 2024-03-22T05:55:36.440526+0000 (oldest deadline 2024-03-22T05:55:59.340383+0000) 2024-03-22T05:56:00.044 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:55:59 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-5[82660]: 2024-03-22T05:55:59.716+0000 7fad9b6cb700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:28.808216+0000 front 2024-03-22T05:55:28.808383+0000 (oldest deadline 2024-03-22T05:55:54.108160+0000) 2024-03-22T05:56:00.044 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:55:59 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:55:59.642+0000 7f53af753700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:25.432637+0000 front 2024-03-22T05:55:25.432503+0000 (oldest deadline 2024-03-22T05:55:50.732484+0000) 2024-03-22T05:56:00.044 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:55:59 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6[87212]: 2024-03-22T05:55:59.766+0000 7f1410de7700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:29.197283+0000 front 2024-03-22T05:55:29.197317+0000 (oldest deadline 2024-03-22T05:55:50.297145+0000) 2024-03-22T05:56:00.045 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:55:59 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6[87212]: 2024-03-22T05:55:59.766+0000 7f1410de7700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.5:6814 osd.1 since back 2024-03-22T05:55:36.698732+0000 front 2024-03-22T05:55:36.698599+0000 (oldest deadline 2024-03-22T05:55:58.998640+0000) 2024-03-22T05:56:00.758 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:00 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:56:00.645+0000 7f53af753700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:25.432637+0000 front 2024-03-22T05:55:25.432503+0000 (oldest deadline 2024-03-22T05:55:50.732484+0000) 2024-03-22T05:56:00.758 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:00 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:56:00.645+0000 7f53af753700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.5:6814 osd.1 since back 2024-03-22T05:55:35.934150+0000 front 2024-03-22T05:55:35.934188+0000 (oldest deadline 2024-03-22T05:56:00.634091+0000) 2024-03-22T05:56:00.758 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 22 05:56:00 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-4[78076]: 2024-03-22T05:56:00.415+0000 7f91c535a700 -1 osd.4 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:27.039500+0000 front 2024-03-22T05:55:27.039424+0000 (oldest deadline 2024-03-22T05:55:51.739358+0000) 2024-03-22T05:56:00.758 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 22 05:56:00 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-4[78076]: 2024-03-22T05:56:00.415+0000 7f91c535a700 -1 osd.4 158 heartbeat_check: no reply from 172.21.15.5:6814 osd.1 since back 2024-03-22T05:55:36.440472+0000 front 2024-03-22T05:55:36.440526+0000 (oldest deadline 2024-03-22T05:55:59.340383+0000) 2024-03-22T05:56:01.044 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:56:00 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-5[82660]: 2024-03-22T05:56:00.760+0000 7fad9b6cb700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:28.808216+0000 front 2024-03-22T05:55:28.808383+0000 (oldest deadline 2024-03-22T05:55:54.108160+0000) 2024-03-22T05:56:01.044 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:56:00 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-5[82660]: 2024-03-22T05:56:00.760+0000 7fad9b6cb700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.5:6814 osd.1 since back 2024-03-22T05:55:35.209158+0000 front 2024-03-22T05:55:35.209059+0000 (oldest deadline 2024-03-22T05:56:00.509053+0000) 2024-03-22T05:56:01.044 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:56:00 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6[87212]: 2024-03-22T05:56:00.757+0000 7f1410de7700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:29.197283+0000 front 2024-03-22T05:55:29.197317+0000 (oldest deadline 2024-03-22T05:55:50.297145+0000) 2024-03-22T05:56:01.044 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:56:00 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6[87212]: 2024-03-22T05:56:00.757+0000 7f1410de7700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.5:6814 osd.1 since back 2024-03-22T05:55:36.698732+0000 front 2024-03-22T05:55:36.698599+0000 (oldest deadline 2024-03-22T05:55:58.998640+0000) 2024-03-22T05:56:01.677 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 22 05:56:01 smithi186 podman[101384]: 2024-03-22 05:56:01.357221876 +0000 UTC m=+5.626907675 container died 8f7619b726f372801ef2cded85833a5a056a0c8a7e3fcc9474923d4bb0fc99ce (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-4, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-22T05:56:02.044 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:01 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:56:01.676+0000 7f53af753700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:25.432637+0000 front 2024-03-22T05:55:25.432503+0000 (oldest deadline 2024-03-22T05:55:50.732484+0000) 2024-03-22T05:56:02.044 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:01 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:56:01.676+0000 7f53af753700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.5:6814 osd.1 since back 2024-03-22T05:55:35.934150+0000 front 2024-03-22T05:55:35.934188+0000 (oldest deadline 2024-03-22T05:56:00.634091+0000) 2024-03-22T05:56:02.044 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:56:01 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-5[82660]: 2024-03-22T05:56:01.756+0000 7fad9b6cb700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:28.808216+0000 front 2024-03-22T05:55:28.808383+0000 (oldest deadline 2024-03-22T05:55:54.108160+0000) 2024-03-22T05:56:02.044 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:56:01 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-5[82660]: 2024-03-22T05:56:01.756+0000 7fad9b6cb700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.5:6814 osd.1 since back 2024-03-22T05:55:35.209158+0000 front 2024-03-22T05:55:35.209059+0000 (oldest deadline 2024-03-22T05:56:00.509053+0000) 2024-03-22T05:56:02.045 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:56:01 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6[87212]: 2024-03-22T05:56:01.774+0000 7f1410de7700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:29.197283+0000 front 2024-03-22T05:55:29.197317+0000 (oldest deadline 2024-03-22T05:55:50.297145+0000) 2024-03-22T05:56:02.045 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:56:01 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6[87212]: 2024-03-22T05:56:01.774+0000 7f1410de7700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.5:6814 osd.1 since back 2024-03-22T05:55:36.698732+0000 front 2024-03-22T05:55:36.698599+0000 (oldest deadline 2024-03-22T05:55:58.998640+0000) 2024-03-22T05:56:02.045 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 22 05:56:01 smithi186 podman[101384]: 2024-03-22 05:56:01.677697814 +0000 UTC m=+5.947383621 container cleanup 8f7619b726f372801ef2cded85833a5a056a0c8a7e3fcc9474923d4bb0fc99ce (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-4, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-22T05:56:02.045 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 22 05:56:01 smithi186 bash[101384]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-4 2024-03-22T05:56:02.045 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 22 05:56:01 smithi186 podman[101401]: 2024-03-22 05:56:01.879210821 +0000 UTC m=+0.516123922 container remove 8f7619b726f372801ef2cded85833a5a056a0c8a7e3fcc9474923d4bb0fc99ce (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-4, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, maintainer=Guillaume Abrioux ) 2024-03-22T05:56:02.354 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 22 05:56:02 smithi186 podman[101489]: 2024-03-22 05:56:02.117057259 +0000 UTC m=+0.017981173 image pull aa461e7c5e7f322b169c789b389d594b994ea7ffcbc9e24e563a3ec9b69bc3c1 quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9 2024-03-22T05:56:02.354 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 22 05:56:02 smithi186 podman[101489]: 2024-03-22 05:56:02.221928572 +0000 UTC m=+0.122852449 container create 54d15202382883a7ba34a3bd303a38f3694ccc23d5750e91810491a918412b5a (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-4-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-22T05:56:02.703 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 22 05:56:02 smithi186 podman[101489]: 2024-03-22 05:56:02.474562921 +0000 UTC m=+0.375486810 container init 54d15202382883a7ba34a3bd303a38f3694ccc23d5750e91810491a918412b5a (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-4-deactivate, GIT_CLEAN=True, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-22T05:56:02.703 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 22 05:56:02 smithi186 podman[101489]: 2024-03-22 05:56:02.478457476 +0000 UTC m=+0.379381359 container start 54d15202382883a7ba34a3bd303a38f3694ccc23d5750e91810491a918412b5a (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-4-deactivate, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD) 2024-03-22T05:56:02.703 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 22 05:56:02 smithi186 podman[101489]: 2024-03-22 05:56:02.582870527 +0000 UTC m=+0.483794440 container attach 54d15202382883a7ba34a3bd303a38f3694ccc23d5750e91810491a918412b5a (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-4-deactivate, maintainer=Guillaume Abrioux , RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-22T05:56:03.044 INFO:journalctl@ceph.osd.4.smithi186.stdout:Mar 22 05:56:02 smithi186 podman[101489]: 2024-03-22 05:56:02.776751407 +0000 UTC m=+0.677675287 container died 54d15202382883a7ba34a3bd303a38f3694ccc23d5750e91810491a918412b5a (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-4-deactivate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-22T05:56:03.044 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:56:02 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-5[82660]: 2024-03-22T05:56:02.740+0000 7fad9b6cb700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:28.808216+0000 front 2024-03-22T05:55:28.808383+0000 (oldest deadline 2024-03-22T05:55:54.108160+0000) 2024-03-22T05:56:03.044 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:56:02 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-5[82660]: 2024-03-22T05:56:02.740+0000 7fad9b6cb700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.5:6814 osd.1 since back 2024-03-22T05:55:35.209158+0000 front 2024-03-22T05:55:35.209059+0000 (oldest deadline 2024-03-22T05:56:00.509053+0000) 2024-03-22T05:56:03.045 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:56:02 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6[87212]: 2024-03-22T05:56:02.810+0000 7f1410de7700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:29.197283+0000 front 2024-03-22T05:55:29.197317+0000 (oldest deadline 2024-03-22T05:55:50.297145+0000) 2024-03-22T05:56:03.045 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:56:02 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6[87212]: 2024-03-22T05:56:02.810+0000 7f1410de7700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.5:6814 osd.1 since back 2024-03-22T05:55:36.698732+0000 front 2024-03-22T05:55:36.698599+0000 (oldest deadline 2024-03-22T05:55:58.998640+0000) 2024-03-22T05:56:03.045 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:02 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:56:02.702+0000 7f53af753700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:25.432637+0000 front 2024-03-22T05:55:25.432503+0000 (oldest deadline 2024-03-22T05:55:50.732484+0000) 2024-03-22T05:56:03.045 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:02 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:56:02.702+0000 7f53af753700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.5:6814 osd.1 since back 2024-03-22T05:55:35.934150+0000 front 2024-03-22T05:55:35.934188+0000 (oldest deadline 2024-03-22T05:56:00.634091+0000) 2024-03-22T05:56:03.449 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@osd.4.service' 2024-03-22T05:56:03.531 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-22T05:56:03.531 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-22T05:56:03.532 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-22T05:56:03.532 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@osd.5 2024-03-22T05:56:03.688 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:56:03 smithi186 systemd[1]: Stopping Ceph osd.5 for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:56:03.689 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:03 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:56:03.675+0000 7f53af753700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:25.432637+0000 front 2024-03-22T05:55:25.432503+0000 (oldest deadline 2024-03-22T05:55:50.732484+0000) 2024-03-22T05:56:03.689 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:03 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:56:03.675+0000 7f53af753700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.5:6814 osd.1 since back 2024-03-22T05:55:35.934150+0000 front 2024-03-22T05:55:35.934188+0000 (oldest deadline 2024-03-22T05:56:00.634091+0000) 2024-03-22T05:56:03.952 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:56:03 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[78315]: ts=2024-03-22T05:56:03.554Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=7 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi005.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.5:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi186.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.186:8443: connect: connection refused" 2024-03-22T05:56:03.953 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:56:03 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[78315]: ts=2024-03-22T05:56:03.555Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi005.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.5:8443: connect: connection refused" 2024-03-22T05:56:03.953 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:56:03 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[78315]: ts=2024-03-22T05:56:03.556Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi186.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.186:8443: connect: connection refused" 2024-03-22T05:56:04.044 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:56:03 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6[87212]: 2024-03-22T05:56:03.810+0000 7f1410de7700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:29.197283+0000 front 2024-03-22T05:55:29.197317+0000 (oldest deadline 2024-03-22T05:55:50.297145+0000) 2024-03-22T05:56:04.044 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:56:03 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6[87212]: 2024-03-22T05:56:03.810+0000 7f1410de7700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.5:6814 osd.1 since back 2024-03-22T05:55:36.698732+0000 front 2024-03-22T05:55:36.698599+0000 (oldest deadline 2024-03-22T05:55:58.998640+0000) 2024-03-22T05:56:04.044 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:56:03 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-5[82660]: 2024-03-22T05:56:03.728+0000 7fad9b6cb700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:28.808216+0000 front 2024-03-22T05:55:28.808383+0000 (oldest deadline 2024-03-22T05:55:54.108160+0000) 2024-03-22T05:56:04.044 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:56:03 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-5[82660]: 2024-03-22T05:56:03.728+0000 7fad9b6cb700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.5:6814 osd.1 since back 2024-03-22T05:55:35.209158+0000 front 2024-03-22T05:55:35.209059+0000 (oldest deadline 2024-03-22T05:56:00.509053+0000) 2024-03-22T05:56:04.044 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:56:03 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-5[82660]: 2024-03-22T05:56:03.847+0000 7fada30df700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T05:56:04.044 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:56:03 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-5[82660]: 2024-03-22T05:56:03.847+0000 7fada30df700 -1 osd.5 158 *** Got signal Terminated *** 2024-03-22T05:56:04.044 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:56:03 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-5[82660]: 2024-03-22T05:56:03.847+0000 7fada30df700 -1 osd.5 158 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-22T05:56:05.044 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:04 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:56:04.705+0000 7f53af753700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:25.432637+0000 front 2024-03-22T05:55:25.432503+0000 (oldest deadline 2024-03-22T05:55:50.732484+0000) 2024-03-22T05:56:05.044 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:04 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:56:04.705+0000 7f53af753700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.5:6814 osd.1 since back 2024-03-22T05:55:35.934150+0000 front 2024-03-22T05:55:35.934188+0000 (oldest deadline 2024-03-22T05:56:00.634091+0000) 2024-03-22T05:56:05.044 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:56:04 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-5[82660]: 2024-03-22T05:56:04.749+0000 7fad9b6cb700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:28.808216+0000 front 2024-03-22T05:55:28.808383+0000 (oldest deadline 2024-03-22T05:55:54.108160+0000) 2024-03-22T05:56:05.044 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:56:04 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-5[82660]: 2024-03-22T05:56:04.749+0000 7fad9b6cb700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.5:6814 osd.1 since back 2024-03-22T05:55:35.209158+0000 front 2024-03-22T05:55:35.209059+0000 (oldest deadline 2024-03-22T05:56:00.509053+0000) 2024-03-22T05:56:05.044 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:56:04 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6[87212]: 2024-03-22T05:56:04.843+0000 7f1410de7700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:29.197283+0000 front 2024-03-22T05:55:29.197317+0000 (oldest deadline 2024-03-22T05:55:50.297145+0000) 2024-03-22T05:56:05.044 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:56:04 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6[87212]: 2024-03-22T05:56:04.843+0000 7f1410de7700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.5:6814 osd.1 since back 2024-03-22T05:55:36.698732+0000 front 2024-03-22T05:55:36.698599+0000 (oldest deadline 2024-03-22T05:55:58.998640+0000) 2024-03-22T05:56:06.044 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:05 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:56:05.673+0000 7f53af753700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:25.432637+0000 front 2024-03-22T05:55:25.432503+0000 (oldest deadline 2024-03-22T05:55:50.732484+0000) 2024-03-22T05:56:06.044 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:05 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:56:05.673+0000 7f53af753700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.5:6814 osd.1 since back 2024-03-22T05:55:35.934150+0000 front 2024-03-22T05:55:35.934188+0000 (oldest deadline 2024-03-22T05:56:00.634091+0000) 2024-03-22T05:56:06.044 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:56:05 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-5[82660]: 2024-03-22T05:56:05.732+0000 7fad9b6cb700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:28.808216+0000 front 2024-03-22T05:55:28.808383+0000 (oldest deadline 2024-03-22T05:55:54.108160+0000) 2024-03-22T05:56:06.044 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:56:05 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-5[82660]: 2024-03-22T05:56:05.732+0000 7fad9b6cb700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.5:6814 osd.1 since back 2024-03-22T05:55:35.209158+0000 front 2024-03-22T05:55:35.209059+0000 (oldest deadline 2024-03-22T05:56:00.509053+0000) 2024-03-22T05:56:06.044 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:56:05 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6[87212]: 2024-03-22T05:56:05.851+0000 7f1410de7700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:29.197283+0000 front 2024-03-22T05:55:29.197317+0000 (oldest deadline 2024-03-22T05:55:50.297145+0000) 2024-03-22T05:56:06.045 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:56:05 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6[87212]: 2024-03-22T05:56:05.851+0000 7f1410de7700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.5:6814 osd.1 since back 2024-03-22T05:55:36.698732+0000 front 2024-03-22T05:55:36.698599+0000 (oldest deadline 2024-03-22T05:55:58.998640+0000) 2024-03-22T05:56:07.044 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:56:06 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6[87212]: 2024-03-22T05:56:06.868+0000 7f1410de7700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:29.197283+0000 front 2024-03-22T05:55:29.197317+0000 (oldest deadline 2024-03-22T05:55:50.297145+0000) 2024-03-22T05:56:07.044 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:56:06 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6[87212]: 2024-03-22T05:56:06.869+0000 7f1410de7700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.5:6814 osd.1 since back 2024-03-22T05:55:36.698732+0000 front 2024-03-22T05:55:36.698599+0000 (oldest deadline 2024-03-22T05:55:58.998640+0000) 2024-03-22T05:56:07.044 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:06 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:56:06.654+0000 7f53af753700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:25.432637+0000 front 2024-03-22T05:55:25.432503+0000 (oldest deadline 2024-03-22T05:55:50.732484+0000) 2024-03-22T05:56:07.044 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:06 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:56:06.654+0000 7f53af753700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.5:6814 osd.1 since back 2024-03-22T05:55:35.934150+0000 front 2024-03-22T05:55:35.934188+0000 (oldest deadline 2024-03-22T05:56:00.634091+0000) 2024-03-22T05:56:07.044 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:56:06 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-5[82660]: 2024-03-22T05:56:06.695+0000 7fad9b6cb700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:28.808216+0000 front 2024-03-22T05:55:28.808383+0000 (oldest deadline 2024-03-22T05:55:54.108160+0000) 2024-03-22T05:56:07.044 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:56:06 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-5[82660]: 2024-03-22T05:56:06.695+0000 7fad9b6cb700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.5:6814 osd.1 since back 2024-03-22T05:55:35.209158+0000 front 2024-03-22T05:55:35.209059+0000 (oldest deadline 2024-03-22T05:56:00.509053+0000) 2024-03-22T05:56:08.044 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:56:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6[87212]: 2024-03-22T05:56:07.847+0000 7f1410de7700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:29.197283+0000 front 2024-03-22T05:55:29.197317+0000 (oldest deadline 2024-03-22T05:55:50.297145+0000) 2024-03-22T05:56:08.044 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:56:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6[87212]: 2024-03-22T05:56:07.847+0000 7f1410de7700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.5:6814 osd.1 since back 2024-03-22T05:55:36.698732+0000 front 2024-03-22T05:55:36.698599+0000 (oldest deadline 2024-03-22T05:55:58.998640+0000) 2024-03-22T05:56:08.044 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:56:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6[87212]: 2024-03-22T05:56:07.847+0000 7f1410de7700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.5:6822 osd.2 since back 2024-03-22T05:55:43.600246+0000 front 2024-03-22T05:55:43.600343+0000 (oldest deadline 2024-03-22T05:56:07.600629+0000) 2024-03-22T05:56:08.044 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:56:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-5[82660]: 2024-03-22T05:56:07.704+0000 7fad9b6cb700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:28.808216+0000 front 2024-03-22T05:55:28.808383+0000 (oldest deadline 2024-03-22T05:55:54.108160+0000) 2024-03-22T05:56:08.044 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:56:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-5[82660]: 2024-03-22T05:56:07.704+0000 7fad9b6cb700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.5:6814 osd.1 since back 2024-03-22T05:55:35.209158+0000 front 2024-03-22T05:55:35.209059+0000 (oldest deadline 2024-03-22T05:56:00.509053+0000) 2024-03-22T05:56:08.045 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:56:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-5[82660]: 2024-03-22T05:56:07.704+0000 7fad9b6cb700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.5:6822 osd.2 since back 2024-03-22T05:55:44.411139+0000 front 2024-03-22T05:55:44.411100+0000 (oldest deadline 2024-03-22T05:56:07.310960+0000) 2024-03-22T05:56:08.045 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:56:07.619+0000 7f53af753700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:25.432637+0000 front 2024-03-22T05:55:25.432503+0000 (oldest deadline 2024-03-22T05:55:50.732484+0000) 2024-03-22T05:56:08.045 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:07 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:56:07.619+0000 7f53af753700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.5:6814 osd.1 since back 2024-03-22T05:55:35.934150+0000 front 2024-03-22T05:55:35.934188+0000 (oldest deadline 2024-03-22T05:56:00.634091+0000) 2024-03-22T05:56:08.872 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:08 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:56:08.588+0000 7f53af753700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:25.432637+0000 front 2024-03-22T05:55:25.432503+0000 (oldest deadline 2024-03-22T05:55:50.732484+0000) 2024-03-22T05:56:08.873 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:08 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:56:08.588+0000 7f53af753700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.5:6814 osd.1 since back 2024-03-22T05:55:35.934150+0000 front 2024-03-22T05:55:35.934188+0000 (oldest deadline 2024-03-22T05:56:00.634091+0000) 2024-03-22T05:56:08.873 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:56:08 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-5[82660]: 2024-03-22T05:56:08.755+0000 7fad9b6cb700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:28.808216+0000 front 2024-03-22T05:55:28.808383+0000 (oldest deadline 2024-03-22T05:55:54.108160+0000) 2024-03-22T05:56:08.873 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:56:08 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-5[82660]: 2024-03-22T05:56:08.755+0000 7fad9b6cb700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.5:6814 osd.1 since back 2024-03-22T05:55:35.209158+0000 front 2024-03-22T05:55:35.209059+0000 (oldest deadline 2024-03-22T05:56:00.509053+0000) 2024-03-22T05:56:08.873 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:56:08 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-5[82660]: 2024-03-22T05:56:08.755+0000 7fad9b6cb700 -1 osd.5 158 heartbeat_check: no reply from 172.21.15.5:6822 osd.2 since back 2024-03-22T05:55:44.411139+0000 front 2024-03-22T05:55:44.411100+0000 (oldest deadline 2024-03-22T05:56:07.310960+0000) 2024-03-22T05:56:09.169 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:56:08 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6[87212]: 2024-03-22T05:56:08.871+0000 7f1410de7700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:29.197283+0000 front 2024-03-22T05:55:29.197317+0000 (oldest deadline 2024-03-22T05:55:50.297145+0000) 2024-03-22T05:56:09.169 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:56:08 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6[87212]: 2024-03-22T05:56:08.871+0000 7f1410de7700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.5:6814 osd.1 since back 2024-03-22T05:55:36.698732+0000 front 2024-03-22T05:55:36.698599+0000 (oldest deadline 2024-03-22T05:55:58.998640+0000) 2024-03-22T05:56:09.169 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:56:08 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6[87212]: 2024-03-22T05:56:08.871+0000 7f1410de7700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.5:6822 osd.2 since back 2024-03-22T05:55:43.600246+0000 front 2024-03-22T05:55:43.600343+0000 (oldest deadline 2024-03-22T05:56:07.600629+0000) 2024-03-22T05:56:09.508 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:56:09 smithi186 podman[101595]: 2024-03-22 05:56:09.251152936 +0000 UTC m=+5.493875497 container died b80f7eceb2d771db9d94f7071eb90160cd2fcf2524b499f5f8733a2f5fbb755d (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True) 2024-03-22T05:56:09.508 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:56:09 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[64327]: ts=2024-03-22T05:56:09.169Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.5:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.5:8765: connect: connection refused" 2024-03-22T05:56:09.508 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:56:09 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[64327]: ts=2024-03-22T05:56:09.172Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.5:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.5:8765: connect: connection refused" 2024-03-22T05:56:09.508 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:56:09 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[64327]: ts=2024-03-22T05:56:09.173Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.5:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.5:8765: connect: connection refused" 2024-03-22T05:56:09.508 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:56:09 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[64327]: ts=2024-03-22T05:56:09.174Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.5:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.5:8765: connect: connection refused" 2024-03-22T05:56:09.508 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:56:09 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[64327]: ts=2024-03-22T05:56:09.174Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.5:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.5:8765: connect: connection refused" 2024-03-22T05:56:09.794 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:56:09 smithi186 podman[101595]: 2024-03-22 05:56:09.706940089 +0000 UTC m=+5.949662635 container cleanup b80f7eceb2d771db9d94f7071eb90160cd2fcf2524b499f5f8733a2f5fbb755d (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-22T05:56:09.794 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:56:09 smithi186 bash[101595]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-5 2024-03-22T05:56:09.794 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:09 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:56:09.608+0000 7f53af753700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:25.432637+0000 front 2024-03-22T05:55:25.432503+0000 (oldest deadline 2024-03-22T05:55:50.732484+0000) 2024-03-22T05:56:09.794 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:09 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:56:09.608+0000 7f53af753700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.5:6814 osd.1 since back 2024-03-22T05:55:35.934150+0000 front 2024-03-22T05:55:35.934188+0000 (oldest deadline 2024-03-22T05:56:00.634091+0000) 2024-03-22T05:56:10.267 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:56:09 smithi186 podman[101627]: 2024-03-22 05:56:09.97518201 +0000 UTC m=+0.723715291 container remove b80f7eceb2d771db9d94f7071eb90160cd2fcf2524b499f5f8733a2f5fbb755d (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-22T05:56:10.267 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:56:09 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6[87212]: 2024-03-22T05:56:09.848+0000 7f1410de7700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:29.197283+0000 front 2024-03-22T05:55:29.197317+0000 (oldest deadline 2024-03-22T05:55:50.297145+0000) 2024-03-22T05:56:10.267 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:56:09 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6[87212]: 2024-03-22T05:56:09.848+0000 7f1410de7700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.5:6814 osd.1 since back 2024-03-22T05:55:36.698732+0000 front 2024-03-22T05:55:36.698599+0000 (oldest deadline 2024-03-22T05:55:58.998640+0000) 2024-03-22T05:56:10.267 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:56:09 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6[87212]: 2024-03-22T05:56:09.848+0000 7f1410de7700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.5:6822 osd.2 since back 2024-03-22T05:55:43.600246+0000 front 2024-03-22T05:55:43.600343+0000 (oldest deadline 2024-03-22T05:56:07.600629+0000) 2024-03-22T05:56:10.544 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:56:10 smithi186 podman[101717]: 2024-03-22 05:56:10.267565249 +0000 UTC m=+0.106778107 container create 00eb28dc10e7900b8b473f0cbbaa969d0cb5cc814fef6f8fd53d23ec0a75f220 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-5-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, GIT_BRANCH=HEAD) 2024-03-22T05:56:10.544 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:56:10 smithi186 podman[101717]: 2024-03-22 05:56:10.179620492 +0000 UTC m=+0.018833364 image pull aa461e7c5e7f322b169c789b389d594b994ea7ffcbc9e24e563a3ec9b69bc3c1 quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9 2024-03-22T05:56:10.544 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:56:10 smithi186 podman[101717]: 2024-03-22 05:56:10.487034184 +0000 UTC m=+0.326247048 container init 00eb28dc10e7900b8b473f0cbbaa969d0cb5cc814fef6f8fd53d23ec0a75f220 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, org.label-schema.schema-version=1.0) 2024-03-22T05:56:10.544 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:56:10 smithi186 podman[101717]: 2024-03-22 05:56:10.490443605 +0000 UTC m=+0.329656472 container start 00eb28dc10e7900b8b473f0cbbaa969d0cb5cc814fef6f8fd53d23ec0a75f220 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-5-deactivate, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319) 2024-03-22T05:56:10.834 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:10 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:56:10.577+0000 7f53af753700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:25.432637+0000 front 2024-03-22T05:55:25.432503+0000 (oldest deadline 2024-03-22T05:55:50.732484+0000) 2024-03-22T05:56:10.840 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:10 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:56:10.577+0000 7f53af753700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.5:6814 osd.1 since back 2024-03-22T05:55:35.934150+0000 front 2024-03-22T05:55:35.934188+0000 (oldest deadline 2024-03-22T05:56:00.634091+0000) 2024-03-22T05:56:10.840 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:10 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:56:10.577+0000 7f53af753700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.5:6822 osd.2 since back 2024-03-22T05:55:44.536258+0000 front 2024-03-22T05:55:44.536234+0000 (oldest deadline 2024-03-22T05:56:09.836105+0000) 2024-03-22T05:56:10.840 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:56:10 smithi186 podman[101717]: 2024-03-22 05:56:10.557733555 +0000 UTC m=+0.396946446 container attach 00eb28dc10e7900b8b473f0cbbaa969d0cb5cc814fef6f8fd53d23ec0a75f220 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-22T05:56:10.841 INFO:journalctl@ceph.osd.5.smithi186.stdout:Mar 22 05:56:10 smithi186 podman[101717]: 2024-03-22 05:56:10.783617864 +0000 UTC m=+0.622830729 container died 00eb28dc10e7900b8b473f0cbbaa969d0cb5cc814fef6f8fd53d23ec0a75f220 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-5-deactivate, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-22T05:56:11.294 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:56:10 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6[87212]: 2024-03-22T05:56:10.833+0000 7f1410de7700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:29.197283+0000 front 2024-03-22T05:55:29.197317+0000 (oldest deadline 2024-03-22T05:55:50.297145+0000) 2024-03-22T05:56:11.294 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:56:10 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6[87212]: 2024-03-22T05:56:10.833+0000 7f1410de7700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.5:6814 osd.1 since back 2024-03-22T05:55:36.698732+0000 front 2024-03-22T05:55:36.698599+0000 (oldest deadline 2024-03-22T05:55:58.998640+0000) 2024-03-22T05:56:11.294 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:56:10 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6[87212]: 2024-03-22T05:56:10.833+0000 7f1410de7700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.5:6822 osd.2 since back 2024-03-22T05:55:43.600246+0000 front 2024-03-22T05:55:43.600343+0000 (oldest deadline 2024-03-22T05:56:07.600629+0000) 2024-03-22T05:56:11.398 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@osd.5.service' 2024-03-22T05:56:11.439 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-22T05:56:11.439 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-22T05:56:11.439 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-22T05:56:11.439 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@osd.6 2024-03-22T05:56:11.794 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:56:11 smithi186 systemd[1]: Stopping Ceph osd.6 for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:56:11.794 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:11 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:56:11.601+0000 7f53af753700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:25.432637+0000 front 2024-03-22T05:55:25.432503+0000 (oldest deadline 2024-03-22T05:55:50.732484+0000) 2024-03-22T05:56:11.794 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:11 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:56:11.601+0000 7f53af753700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.5:6814 osd.1 since back 2024-03-22T05:55:35.934150+0000 front 2024-03-22T05:55:35.934188+0000 (oldest deadline 2024-03-22T05:56:00.634091+0000) 2024-03-22T05:56:11.795 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:11 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:56:11.601+0000 7f53af753700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.5:6822 osd.2 since back 2024-03-22T05:55:44.536258+0000 front 2024-03-22T05:55:44.536234+0000 (oldest deadline 2024-03-22T05:56:09.836105+0000) 2024-03-22T05:56:12.294 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:56:11 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6[87212]: 2024-03-22T05:56:11.826+0000 7f14187fb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T05:56:12.294 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:56:11 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6[87212]: 2024-03-22T05:56:11.826+0000 7f14187fb700 -1 osd.6 158 *** Got signal Terminated *** 2024-03-22T05:56:12.294 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:56:11 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6[87212]: 2024-03-22T05:56:11.826+0000 7f14187fb700 -1 osd.6 158 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-22T05:56:12.294 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:56:11 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6[87212]: 2024-03-22T05:56:11.838+0000 7f1410de7700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:29.197283+0000 front 2024-03-22T05:55:29.197317+0000 (oldest deadline 2024-03-22T05:55:50.297145+0000) 2024-03-22T05:56:12.294 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:56:11 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6[87212]: 2024-03-22T05:56:11.838+0000 7f1410de7700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.5:6814 osd.1 since back 2024-03-22T05:55:36.698732+0000 front 2024-03-22T05:55:36.698599+0000 (oldest deadline 2024-03-22T05:55:58.998640+0000) 2024-03-22T05:56:12.295 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:56:11 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6[87212]: 2024-03-22T05:56:11.838+0000 7f1410de7700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.5:6822 osd.2 since back 2024-03-22T05:55:43.600246+0000 front 2024-03-22T05:55:43.600343+0000 (oldest deadline 2024-03-22T05:56:07.600629+0000) 2024-03-22T05:56:12.881 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:12 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:56:12.588+0000 7f53af753700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:25.432637+0000 front 2024-03-22T05:55:25.432503+0000 (oldest deadline 2024-03-22T05:55:50.732484+0000) 2024-03-22T05:56:12.882 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:12 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:56:12.588+0000 7f53af753700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.5:6814 osd.1 since back 2024-03-22T05:55:35.934150+0000 front 2024-03-22T05:55:35.934188+0000 (oldest deadline 2024-03-22T05:56:00.634091+0000) 2024-03-22T05:56:12.882 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:12 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:56:12.588+0000 7f53af753700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.5:6822 osd.2 since back 2024-03-22T05:55:44.536258+0000 front 2024-03-22T05:55:44.536234+0000 (oldest deadline 2024-03-22T05:56:09.836105+0000) 2024-03-22T05:56:13.294 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:56:12 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6[87212]: 2024-03-22T05:56:12.881+0000 7f1410de7700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:29.197283+0000 front 2024-03-22T05:55:29.197317+0000 (oldest deadline 2024-03-22T05:55:50.297145+0000) 2024-03-22T05:56:13.294 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:56:12 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6[87212]: 2024-03-22T05:56:12.881+0000 7f1410de7700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.5:6814 osd.1 since back 2024-03-22T05:55:36.698732+0000 front 2024-03-22T05:55:36.698599+0000 (oldest deadline 2024-03-22T05:55:58.998640+0000) 2024-03-22T05:56:13.294 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:56:12 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6[87212]: 2024-03-22T05:56:12.881+0000 7f1410de7700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.5:6822 osd.2 since back 2024-03-22T05:55:43.600246+0000 front 2024-03-22T05:55:43.600343+0000 (oldest deadline 2024-03-22T05:56:07.600629+0000) 2024-03-22T05:56:13.913 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:13 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:56:13.598+0000 7f53af753700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:25.432637+0000 front 2024-03-22T05:55:25.432503+0000 (oldest deadline 2024-03-22T05:55:50.732484+0000) 2024-03-22T05:56:13.913 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:13 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:56:13.598+0000 7f53af753700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.5:6814 osd.1 since back 2024-03-22T05:55:35.934150+0000 front 2024-03-22T05:55:35.934188+0000 (oldest deadline 2024-03-22T05:56:00.634091+0000) 2024-03-22T05:56:13.913 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:13 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:56:13.598+0000 7f53af753700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.5:6822 osd.2 since back 2024-03-22T05:55:44.536258+0000 front 2024-03-22T05:55:44.536234+0000 (oldest deadline 2024-03-22T05:56:09.836105+0000) 2024-03-22T05:56:13.952 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:56:13 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[78315]: ts=2024-03-22T05:56:13.555Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=8 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi186.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.186:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi005.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.5:8443: connect: connection refused" 2024-03-22T05:56:13.952 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:56:13 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[78315]: ts=2024-03-22T05:56:13.555Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi005.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.5:8443: connect: connection refused" 2024-03-22T05:56:13.952 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:56:13 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[78315]: ts=2024-03-22T05:56:13.556Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi186.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.186:8443: connect: connection refused" 2024-03-22T05:56:14.294 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:56:13 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6[87212]: 2024-03-22T05:56:13.912+0000 7f1410de7700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:29.197283+0000 front 2024-03-22T05:55:29.197317+0000 (oldest deadline 2024-03-22T05:55:50.297145+0000) 2024-03-22T05:56:14.294 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:56:13 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6[87212]: 2024-03-22T05:56:13.912+0000 7f1410de7700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.5:6814 osd.1 since back 2024-03-22T05:55:36.698732+0000 front 2024-03-22T05:55:36.698599+0000 (oldest deadline 2024-03-22T05:55:58.998640+0000) 2024-03-22T05:56:14.294 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:56:13 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6[87212]: 2024-03-22T05:56:13.912+0000 7f1410de7700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.5:6822 osd.2 since back 2024-03-22T05:55:43.600246+0000 front 2024-03-22T05:55:43.600343+0000 (oldest deadline 2024-03-22T05:56:07.600629+0000) 2024-03-22T05:56:14.883 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:14 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:56:14.592+0000 7f53af753700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:25.432637+0000 front 2024-03-22T05:55:25.432503+0000 (oldest deadline 2024-03-22T05:55:50.732484+0000) 2024-03-22T05:56:14.884 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:14 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:56:14.592+0000 7f53af753700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.5:6814 osd.1 since back 2024-03-22T05:55:35.934150+0000 front 2024-03-22T05:55:35.934188+0000 (oldest deadline 2024-03-22T05:56:00.634091+0000) 2024-03-22T05:56:14.884 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:14 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:56:14.592+0000 7f53af753700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.5:6822 osd.2 since back 2024-03-22T05:55:44.536258+0000 front 2024-03-22T05:55:44.536234+0000 (oldest deadline 2024-03-22T05:56:09.836105+0000) 2024-03-22T05:56:15.294 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:56:14 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6[87212]: 2024-03-22T05:56:14.882+0000 7f1410de7700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:29.197283+0000 front 2024-03-22T05:55:29.197317+0000 (oldest deadline 2024-03-22T05:55:50.297145+0000) 2024-03-22T05:56:15.294 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:56:14 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6[87212]: 2024-03-22T05:56:14.882+0000 7f1410de7700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.5:6814 osd.1 since back 2024-03-22T05:55:36.698732+0000 front 2024-03-22T05:55:36.698599+0000 (oldest deadline 2024-03-22T05:55:58.998640+0000) 2024-03-22T05:56:15.294 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:56:14 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6[87212]: 2024-03-22T05:56:14.883+0000 7f1410de7700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.5:6822 osd.2 since back 2024-03-22T05:55:43.600246+0000 front 2024-03-22T05:55:43.600343+0000 (oldest deadline 2024-03-22T05:56:07.600629+0000) 2024-03-22T05:56:15.910 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:15 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:56:15.626+0000 7f53af753700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:25.432637+0000 front 2024-03-22T05:55:25.432503+0000 (oldest deadline 2024-03-22T05:55:50.732484+0000) 2024-03-22T05:56:15.910 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:15 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:56:15.626+0000 7f53af753700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.5:6814 osd.1 since back 2024-03-22T05:55:35.934150+0000 front 2024-03-22T05:55:35.934188+0000 (oldest deadline 2024-03-22T05:56:00.634091+0000) 2024-03-22T05:56:15.910 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:15 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:56:15.626+0000 7f53af753700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.5:6822 osd.2 since back 2024-03-22T05:55:44.536258+0000 front 2024-03-22T05:55:44.536234+0000 (oldest deadline 2024-03-22T05:56:09.836105+0000) 2024-03-22T05:56:16.294 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:56:15 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6[87212]: 2024-03-22T05:56:15.909+0000 7f1410de7700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:29.197283+0000 front 2024-03-22T05:55:29.197317+0000 (oldest deadline 2024-03-22T05:55:50.297145+0000) 2024-03-22T05:56:16.294 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:56:15 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6[87212]: 2024-03-22T05:56:15.909+0000 7f1410de7700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.5:6814 osd.1 since back 2024-03-22T05:55:36.698732+0000 front 2024-03-22T05:55:36.698599+0000 (oldest deadline 2024-03-22T05:55:58.998640+0000) 2024-03-22T05:56:16.294 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:56:15 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6[87212]: 2024-03-22T05:56:15.909+0000 7f1410de7700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.5:6822 osd.2 since back 2024-03-22T05:55:43.600246+0000 front 2024-03-22T05:55:43.600343+0000 (oldest deadline 2024-03-22T05:56:07.600629+0000) 2024-03-22T05:56:16.294 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:56:15 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6[87212]: 2024-03-22T05:56:15.909+0000 7f1410de7700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.5:6830 osd.3 since back 2024-03-22T05:55:51.701588+0000 front 2024-03-22T05:55:51.701540+0000 (oldest deadline 2024-03-22T05:56:15.201508+0000) 2024-03-22T05:56:17.044 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:16 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:56:16.661+0000 7f53af753700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:25.432637+0000 front 2024-03-22T05:55:25.432503+0000 (oldest deadline 2024-03-22T05:55:50.732484+0000) 2024-03-22T05:56:17.044 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:16 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:56:16.661+0000 7f53af753700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.5:6814 osd.1 since back 2024-03-22T05:55:35.934150+0000 front 2024-03-22T05:55:35.934188+0000 (oldest deadline 2024-03-22T05:56:00.634091+0000) 2024-03-22T05:56:17.044 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:16 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:56:16.661+0000 7f53af753700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.5:6822 osd.2 since back 2024-03-22T05:55:44.536258+0000 front 2024-03-22T05:55:44.536234+0000 (oldest deadline 2024-03-22T05:56:09.836105+0000) 2024-03-22T05:56:17.544 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:56:17 smithi186 podman[101821]: 2024-03-22 05:56:17.250631663 +0000 UTC m=+5.594192281 container died db7c81962c65d77d151a0f80f2a64c13f9bc914764962ce08157992681ee2cac (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-03-22T05:56:17.886 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:56:17 smithi186 podman[101821]: 2024-03-22 05:56:17.618767091 +0000 UTC m=+5.962327753 container cleanup db7c81962c65d77d151a0f80f2a64c13f9bc914764962ce08157992681ee2cac (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-22T05:56:17.886 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:56:17 smithi186 bash[101821]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6 2024-03-22T05:56:17.886 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:17 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:56:17.648+0000 7f53af753700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:25.432637+0000 front 2024-03-22T05:55:25.432503+0000 (oldest deadline 2024-03-22T05:55:50.732484+0000) 2024-03-22T05:56:17.886 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:17 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:56:17.648+0000 7f53af753700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.5:6814 osd.1 since back 2024-03-22T05:55:35.934150+0000 front 2024-03-22T05:55:35.934188+0000 (oldest deadline 2024-03-22T05:56:00.634091+0000) 2024-03-22T05:56:17.886 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:17 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:56:17.648+0000 7f53af753700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.5:6822 osd.2 since back 2024-03-22T05:55:44.536258+0000 front 2024-03-22T05:55:44.536234+0000 (oldest deadline 2024-03-22T05:56:09.836105+0000) 2024-03-22T05:56:18.240 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:56:17 smithi186 podman[101838]: 2024-03-22 05:56:17.88661555 +0000 UTC m=+0.629792347 container remove db7c81962c65d77d151a0f80f2a64c13f9bc914764962ce08157992681ee2cac (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-22T05:56:18.531 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:56:18 smithi186 podman[101930]: 2024-03-22 05:56:18.142930869 +0000 UTC m=+0.017868881 image pull aa461e7c5e7f322b169c789b389d594b994ea7ffcbc9e24e563a3ec9b69bc3c1 quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9 2024-03-22T05:56:18.532 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:56:18 smithi186 podman[101930]: 2024-03-22 05:56:18.29603446 +0000 UTC m=+0.170972462 container create 0f61221d0e4e7db5abbd336ea76b0b1c7e9192660daaabb95d615f14b1900753 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-22T05:56:18.532 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:56:18 smithi186 podman[101930]: 2024-03-22 05:56:18.532188316 +0000 UTC m=+0.407126328 container init 0f61221d0e4e7db5abbd336ea76b0b1c7e9192660daaabb95d615f14b1900753 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217) 2024-03-22T05:56:18.794 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:18 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:56:18.652+0000 7f53af753700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:25.432637+0000 front 2024-03-22T05:55:25.432503+0000 (oldest deadline 2024-03-22T05:55:50.732484+0000) 2024-03-22T05:56:18.794 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:18 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:56:18.652+0000 7f53af753700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.5:6814 osd.1 since back 2024-03-22T05:55:35.934150+0000 front 2024-03-22T05:55:35.934188+0000 (oldest deadline 2024-03-22T05:56:00.634091+0000) 2024-03-22T05:56:18.794 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:18 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:56:18.652+0000 7f53af753700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.5:6822 osd.2 since back 2024-03-22T05:55:44.536258+0000 front 2024-03-22T05:55:44.536234+0000 (oldest deadline 2024-03-22T05:56:09.836105+0000) 2024-03-22T05:56:18.794 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:18 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:56:18.652+0000 7f53af753700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.5:6830 osd.3 since back 2024-03-22T05:55:49.836693+0000 front 2024-03-22T05:55:49.836682+0000 (oldest deadline 2024-03-22T05:56:18.037087+0000) 2024-03-22T05:56:18.794 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:56:18 smithi186 podman[101930]: 2024-03-22 05:56:18.53536603 +0000 UTC m=+0.410304045 container start 0f61221d0e4e7db5abbd336ea76b0b1c7e9192660daaabb95d615f14b1900753 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-22T05:56:18.794 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:56:18 smithi186 podman[101930]: 2024-03-22 05:56:18.623780925 +0000 UTC m=+0.498718947 container attach 0f61221d0e4e7db5abbd336ea76b0b1c7e9192660daaabb95d615f14b1900753 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6-deactivate, org.label-schema.build-date=20240319, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-22T05:56:19.118 INFO:journalctl@ceph.osd.6.smithi186.stdout:Mar 22 05:56:18 smithi186 podman[101930]: 2024-03-22 05:56:18.828331631 +0000 UTC m=+0.703269633 container died 0f61221d0e4e7db5abbd336ea76b0b1c7e9192660daaabb95d615f14b1900753 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, io.buildah.version=1.33.5, ceph=True) 2024-03-22T05:56:19.400 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@osd.6.service' 2024-03-22T05:56:19.440 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-22T05:56:19.440 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-22T05:56:19.440 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-22T05:56:19.440 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@osd.7 2024-03-22T05:56:19.659 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:19 smithi186 systemd[1]: Stopping Ceph osd.7 for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:56:20.044 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:19 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:56:19.659+0000 7f53af753700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:25.432637+0000 front 2024-03-22T05:55:25.432503+0000 (oldest deadline 2024-03-22T05:55:50.732484+0000) 2024-03-22T05:56:20.044 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:19 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:56:19.659+0000 7f53af753700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.5:6814 osd.1 since back 2024-03-22T05:55:35.934150+0000 front 2024-03-22T05:55:35.934188+0000 (oldest deadline 2024-03-22T05:56:00.634091+0000) 2024-03-22T05:56:20.044 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:19 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:56:19.659+0000 7f53af753700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.5:6822 osd.2 since back 2024-03-22T05:55:44.536258+0000 front 2024-03-22T05:55:44.536234+0000 (oldest deadline 2024-03-22T05:56:09.836105+0000) 2024-03-22T05:56:20.044 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:19 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:56:19.659+0000 7f53af753700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.5:6830 osd.3 since back 2024-03-22T05:55:49.836693+0000 front 2024-03-22T05:55:49.836682+0000 (oldest deadline 2024-03-22T05:56:18.037087+0000) 2024-03-22T05:56:20.044 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:19 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:56:19.754+0000 7f53b315f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-22T05:56:20.045 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:19 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:56:19.754+0000 7f53b315f700 -1 osd.7 158 *** Got signal Terminated *** 2024-03-22T05:56:20.045 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:19 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:56:19.754+0000 7f53b315f700 -1 osd.7 158 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-22T05:56:21.044 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:20 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:56:20.650+0000 7f53af753700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:25.432637+0000 front 2024-03-22T05:55:25.432503+0000 (oldest deadline 2024-03-22T05:55:50.732484+0000) 2024-03-22T05:56:21.044 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:20 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:56:20.650+0000 7f53af753700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.5:6814 osd.1 since back 2024-03-22T05:55:35.934150+0000 front 2024-03-22T05:55:35.934188+0000 (oldest deadline 2024-03-22T05:56:00.634091+0000) 2024-03-22T05:56:21.044 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:20 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:56:20.650+0000 7f53af753700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.5:6822 osd.2 since back 2024-03-22T05:55:44.536258+0000 front 2024-03-22T05:55:44.536234+0000 (oldest deadline 2024-03-22T05:56:09.836105+0000) 2024-03-22T05:56:21.044 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:20 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:56:20.650+0000 7f53af753700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.5:6830 osd.3 since back 2024-03-22T05:55:49.836693+0000 front 2024-03-22T05:55:49.836682+0000 (oldest deadline 2024-03-22T05:56:18.037087+0000) 2024-03-22T05:56:22.044 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:21 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:56:21.679+0000 7f53af753700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:25.432637+0000 front 2024-03-22T05:55:25.432503+0000 (oldest deadline 2024-03-22T05:55:50.732484+0000) 2024-03-22T05:56:22.044 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:21 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:56:21.679+0000 7f53af753700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.5:6814 osd.1 since back 2024-03-22T05:55:35.934150+0000 front 2024-03-22T05:55:35.934188+0000 (oldest deadline 2024-03-22T05:56:00.634091+0000) 2024-03-22T05:56:22.044 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:21 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:56:21.679+0000 7f53af753700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.5:6822 osd.2 since back 2024-03-22T05:55:44.536258+0000 front 2024-03-22T05:55:44.536234+0000 (oldest deadline 2024-03-22T05:56:09.836105+0000) 2024-03-22T05:56:22.044 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:21 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:56:21.679+0000 7f53af753700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.5:6830 osd.3 since back 2024-03-22T05:55:49.836693+0000 front 2024-03-22T05:55:49.836682+0000 (oldest deadline 2024-03-22T05:56:18.037087+0000) 2024-03-22T05:56:23.044 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:22 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:56:22.637+0000 7f53af753700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:25.432637+0000 front 2024-03-22T05:55:25.432503+0000 (oldest deadline 2024-03-22T05:55:50.732484+0000) 2024-03-22T05:56:23.044 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:22 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:56:22.637+0000 7f53af753700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.5:6814 osd.1 since back 2024-03-22T05:55:35.934150+0000 front 2024-03-22T05:55:35.934188+0000 (oldest deadline 2024-03-22T05:56:00.634091+0000) 2024-03-22T05:56:23.044 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:22 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:56:22.637+0000 7f53af753700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.5:6822 osd.2 since back 2024-03-22T05:55:44.536258+0000 front 2024-03-22T05:55:44.536234+0000 (oldest deadline 2024-03-22T05:56:09.836105+0000) 2024-03-22T05:56:23.044 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:22 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:56:22.637+0000 7f53af753700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.5:6830 osd.3 since back 2024-03-22T05:55:49.836693+0000 front 2024-03-22T05:55:49.836682+0000 (oldest deadline 2024-03-22T05:56:18.037087+0000) 2024-03-22T05:56:23.862 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:56:23 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[78315]: ts=2024-03-22T05:56:23.556Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=8 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi186.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.186:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi005.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.5:8443: connect: connection refused" 2024-03-22T05:56:23.862 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:56:23 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[78315]: ts=2024-03-22T05:56:23.556Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi005.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.5:8443: connect: connection refused" 2024-03-22T05:56:23.862 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:56:23 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[78315]: ts=2024-03-22T05:56:23.557Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi186.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.186:8443: connect: connection refused" 2024-03-22T05:56:24.044 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:23 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:56:23.594+0000 7f53af753700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:25.432637+0000 front 2024-03-22T05:55:25.432503+0000 (oldest deadline 2024-03-22T05:55:50.732484+0000) 2024-03-22T05:56:24.044 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:23 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:56:23.594+0000 7f53af753700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.5:6814 osd.1 since back 2024-03-22T05:55:35.934150+0000 front 2024-03-22T05:55:35.934188+0000 (oldest deadline 2024-03-22T05:56:00.634091+0000) 2024-03-22T05:56:24.044 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:23 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:56:23.594+0000 7f53af753700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.5:6822 osd.2 since back 2024-03-22T05:55:44.536258+0000 front 2024-03-22T05:55:44.536234+0000 (oldest deadline 2024-03-22T05:56:09.836105+0000) 2024-03-22T05:56:24.044 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:23 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:56:23.594+0000 7f53af753700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.5:6830 osd.3 since back 2024-03-22T05:55:49.836693+0000 front 2024-03-22T05:55:49.836682+0000 (oldest deadline 2024-03-22T05:56:18.037087+0000) 2024-03-22T05:56:24.044 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:23 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:56:23.594+0000 7f53af753700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.186:6804 osd.4 since back 2024-03-22T05:55:58.037650+0000 front 2024-03-22T05:55:58.037590+0000 (oldest deadline 2024-03-22T05:56:22.737621+0000) 2024-03-22T05:56:25.044 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:24 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:56:24.568+0000 7f53af753700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.5:6806 osd.0 since back 2024-03-22T05:55:25.432637+0000 front 2024-03-22T05:55:25.432503+0000 (oldest deadline 2024-03-22T05:55:50.732484+0000) 2024-03-22T05:56:25.044 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:24 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:56:24.568+0000 7f53af753700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.5:6814 osd.1 since back 2024-03-22T05:55:35.934150+0000 front 2024-03-22T05:55:35.934188+0000 (oldest deadline 2024-03-22T05:56:00.634091+0000) 2024-03-22T05:56:25.044 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:24 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:56:24.568+0000 7f53af753700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.5:6822 osd.2 since back 2024-03-22T05:55:44.536258+0000 front 2024-03-22T05:55:44.536234+0000 (oldest deadline 2024-03-22T05:56:09.836105+0000) 2024-03-22T05:56:25.044 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:24 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:56:24.568+0000 7f53af753700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.5:6830 osd.3 since back 2024-03-22T05:55:49.836693+0000 front 2024-03-22T05:55:49.836682+0000 (oldest deadline 2024-03-22T05:56:18.037087+0000) 2024-03-22T05:56:25.044 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:24 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7[92155]: 2024-03-22T05:56:24.568+0000 7f53af753700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.186:6804 osd.4 since back 2024-03-22T05:55:58.037650+0000 front 2024-03-22T05:55:58.037590+0000 (oldest deadline 2024-03-22T05:56:22.737621+0000) 2024-03-22T05:56:25.388 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:25 smithi186 podman[102035]: 2024-03-22 05:56:25.09019919 +0000 UTC m=+5.435908713 container died fcb46ba843ae84db4a12ed92a080112ef8c5ff9eee34fbe202f32c91a27f2c3d (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-03-22T05:56:25.794 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:25 smithi186 podman[102035]: 2024-03-22 05:56:25.554538623 +0000 UTC m=+5.900248137 container cleanup fcb46ba843ae84db4a12ed92a080112ef8c5ff9eee34fbe202f32c91a27f2c3d (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True) 2024-03-22T05:56:25.794 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:25 smithi186 bash[102035]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7 2024-03-22T05:56:26.173 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:25 smithi186 podman[102053]: 2024-03-22 05:56:25.847680293 +0000 UTC m=+0.749107671 container remove fcb46ba843ae84db4a12ed92a080112ef8c5ff9eee34fbe202f32c91a27f2c3d (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-22T05:56:26.459 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:26 smithi186 podman[102146]: 2024-03-22 05:56:26.173510585 +0000 UTC m=+0.106828878 container create 9618fcd3efa9e7fb246e312411b820a1f61e2a0cb3ac1be9eaf665b2c1c9f8d7 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-22T05:56:26.459 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:26 smithi186 podman[102146]: 2024-03-22 05:56:26.08437491 +0000 UTC m=+0.017693183 image pull aa461e7c5e7f322b169c789b389d594b994ea7ffcbc9e24e563a3ec9b69bc3c1 quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9 2024-03-22T05:56:26.459 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:26 smithi186 podman[102146]: 2024-03-22 05:56:26.40090795 +0000 UTC m=+0.334226211 container init 9618fcd3efa9e7fb246e312411b820a1f61e2a0cb3ac1be9eaf665b2c1c9f8d7 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7-deactivate, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-22T05:56:26.459 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:26 smithi186 podman[102146]: 2024-03-22 05:56:26.404145838 +0000 UTC m=+0.337464088 container start 9618fcd3efa9e7fb246e312411b820a1f61e2a0cb3ac1be9eaf665b2c1c9f8d7 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7-deactivate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240319, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-22T05:56:26.459 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:26 smithi186 podman[102146]: 2024-03-22 05:56:26.458923262 +0000 UTC m=+0.392241535 container attach 9618fcd3efa9e7fb246e312411b820a1f61e2a0cb3ac1be9eaf665b2c1c9f8d7 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7-deactivate, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-22T05:56:26.794 INFO:journalctl@ceph.osd.7.smithi186.stdout:Mar 22 05:56:26 smithi186 podman[102146]: 2024-03-22 05:56:26.693266532 +0000 UTC m=+0.626584783 container died 9618fcd3efa9e7fb246e312411b820a1f61e2a0cb3ac1be9eaf665b2c1c9f8d7 (image=quay.ceph.io/ceph-ci/ceph@sha256:86a7734db939dbe0061161a071f862ec15f51e06c205e37a9b0f00b8755315a9, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-osd-7-deactivate, GIT_BRANCH=HEAD, RELEASE=wip-phlogistonjohn-testing-2024-03-21-1835-ea01217, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-22T05:56:27.294 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@osd.7.service' 2024-03-22T05:56:27.334 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-22T05:56:27.334 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-22T05:56:27.335 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-03-22T05:56:27.335 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@prometheus.a 2024-03-22T05:56:27.544 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:56:27 smithi186 systemd[1]: Stopping Ceph prometheus.a for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:56:27.957 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:56:27 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[64327]: ts=2024-03-22T05:56:27.648Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-22T05:56:27.957 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:56:27 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[64327]: ts=2024-03-22T05:56:27.649Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-03-22T05:56:27.957 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:56:27 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[64327]: ts=2024-03-22T05:56:27.649Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-03-22T05:56:27.957 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:56:27 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[64327]: ts=2024-03-22T05:56:27.649Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-22T05:56:27.957 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:56:27 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[64327]: ts=2024-03-22T05:56:27.649Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-03-22T05:56:27.957 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:56:27 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[64327]: ts=2024-03-22T05:56:27.649Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-03-22T05:56:27.957 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:56:27 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[64327]: ts=2024-03-22T05:56:27.649Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-03-22T05:56:27.957 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:56:27 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[64327]: ts=2024-03-22T05:56:27.649Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-03-22T05:56:27.957 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:56:27 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[64327]: ts=2024-03-22T05:56:27.649Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-03-22T05:56:27.958 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:56:27 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[64327]: ts=2024-03-22T05:56:27.649Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-03-22T05:56:27.958 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:56:27 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[64327]: ts=2024-03-22T05:56:27.649Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-03-22T05:56:27.958 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:56:27 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a[64327]: ts=2024-03-22T05:56:27.649Z caller=main.go:1147 level=info msg="See you next time!" 2024-03-22T05:56:27.958 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:56:27 smithi186 podman[102251]: 2024-03-22 05:56:27.659031749 +0000 UTC m=+0.106561010 container died a5a043588d6dee234004c0e728608ee28c6c52f74391f789223f2eb6244d7dc8 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-22T05:56:28.265 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:56:28 smithi186 podman[102251]: 2024-03-22 05:56:28.048575944 +0000 UTC m=+0.496105202 container cleanup a5a043588d6dee234004c0e728608ee28c6c52f74391f789223f2eb6244d7dc8 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-22T05:56:28.265 INFO:journalctl@ceph.prometheus.a.smithi186.stdout:Mar 22 05:56:28 smithi186 bash[102251]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-prometheus-a 2024-03-22T05:56:28.402 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@prometheus.a.service' 2024-03-22T05:56:28.438 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-22T05:56:28.438 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-03-22T05:56:28.438 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 --force --keep-logs 2024-03-22T05:56:30.194 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:56:29 smithi005 systemd[1]: Stopping Ceph node-exporter.a for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:56:30.452 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:56:30 smithi005 podman[138234]: 2024-03-22 05:56:30.194342932 +0000 UTC m=+0.138878926 container died f635cd72b9d3d88c7ae61ac71a116adc9547b806cdaaa5c33d083cccc0df5576 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-22T05:56:30.952 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:56:30 smithi005 podman[138234]: 2024-03-22 05:56:30.758603443 +0000 UTC m=+0.703139431 container cleanup f635cd72b9d3d88c7ae61ac71a116adc9547b806cdaaa5c33d083cccc0df5576 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-22T05:56:30.952 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:56:30 smithi005 bash[138234]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a 2024-03-22T05:56:31.353 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:56:31 smithi005 podman[138248]: 2024-03-22 05:56:31.043244836 +0000 UTC m=+0.853513288 container remove f635cd72b9d3d88c7ae61ac71a116adc9547b806cdaaa5c33d083cccc0df5576 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-22T05:56:31.353 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:56:31 smithi005 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-03-22T05:56:31.353 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:56:31 smithi005 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-03-22T05:56:31.354 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:56:31 smithi005 systemd[1]: Stopped Ceph node-exporter.a for f59f36e2-e80a-11ee-95cd-87774f69a715. 2024-03-22T05:56:31.354 INFO:journalctl@ceph.node-exporter.a.smithi005.stdout:Mar 22 05:56:31 smithi005 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@node-exporter.a.service: Consumed 3.216s CPU time. 2024-03-22T05:56:31.636 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:56:31 smithi005 systemd[1]: Stopping Ceph alertmanager.a for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:56:31.636 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:56:31 smithi005 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a[78315]: ts=2024-03-22T05:56:31.595Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-22T05:56:31.952 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:56:31 smithi005 podman[138356]: 2024-03-22 05:56:31.636936315 +0000 UTC m=+0.145683381 container died ee99dfc435e88501861d7a0d993f48ebb4ba59cba6956e9c8c82783e92ddad8e (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-22T05:56:32.230 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:56:31 smithi005 podman[138356]: 2024-03-22 05:56:31.954369597 +0000 UTC m=+0.463116661 container cleanup ee99dfc435e88501861d7a0d993f48ebb4ba59cba6956e9c8c82783e92ddad8e (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-22T05:56:32.230 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:56:31 smithi005 bash[138356]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a 2024-03-22T05:56:32.230 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:56:32 smithi005 podman[138369]: 2024-03-22 05:56:32.155424897 +0000 UTC m=+0.513486823 container remove ee99dfc435e88501861d7a0d993f48ebb4ba59cba6956e9c8c82783e92ddad8e (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-22T05:56:32.230 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:56:32 smithi005 podman[138369]: 2024-03-22 05:56:32.230881769 +0000 UTC m=+0.588943650 volume remove c10189d75c8a6b7410728eaaea072e6f92a48ffb533d17bd5fbc413f813faf77 2024-03-22T05:56:32.542 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:56:32 smithi005 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-22T05:56:32.542 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:56:32 smithi005 systemd[1]: Stopped Ceph alertmanager.a for f59f36e2-e80a-11ee-95cd-87774f69a715. 2024-03-22T05:56:32.542 INFO:journalctl@ceph.alertmanager.a.smithi005.stdout:Mar 22 05:56:32 smithi005 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@alertmanager.a.service: Consumed 1.612s CPU time. 2024-03-22T05:56:55.533 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 --force --keep-logs 2024-03-22T05:56:57.377 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:56:57 smithi186 systemd[1]: Stopping Ceph node-exporter.b for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:56:57.634 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:56:57 smithi186 podman[102660]: 2024-03-22 05:56:57.377618881 +0000 UTC m=+0.109667464 container died d7e0b3d201918d57c0675d79f678adb3a4c4d5a8cc2059bf7cdffb53411aa321 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-22T05:56:57.984 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:56:57 smithi186 podman[102660]: 2024-03-22 05:56:57.808512376 +0000 UTC m=+0.540560951 container cleanup d7e0b3d201918d57c0675d79f678adb3a4c4d5a8cc2059bf7cdffb53411aa321 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-22T05:56:57.984 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:56:57 smithi186 bash[102660]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b 2024-03-22T05:56:58.294 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:56:57 smithi186 podman[102673]: 2024-03-22 05:56:57.985107647 +0000 UTC m=+0.609646100 container remove d7e0b3d201918d57c0675d79f678adb3a4c4d5a8cc2059bf7cdffb53411aa321 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-22T05:56:58.294 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:56:57 smithi186 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-03-22T05:56:58.294 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:56:58 smithi186 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-03-22T05:56:58.294 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:56:58 smithi186 systemd[1]: Stopped Ceph node-exporter.b for f59f36e2-e80a-11ee-95cd-87774f69a715. 2024-03-22T05:56:58.294 INFO:journalctl@ceph.node-exporter.b.smithi186.stdout:Mar 22 05:56:58 smithi186 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@node-exporter.b.service: Consumed 3.120s CPU time. 2024-03-22T05:56:58.639 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:56:58 smithi186 systemd[1]: Stopping Ceph grafana.a for f59f36e2-e80a-11ee-95cd-87774f69a715... 2024-03-22T05:56:58.639 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:56:58 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=server t=2024-03-22T05:56:58.629784508Z level=info msg="Shutdown started" reason="System signal: terminated" 2024-03-22T05:56:58.639 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:56:58 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=ticker t=2024-03-22T05:56:58.630134396Z level=info msg=stopped last_tick=2024-03-22T05:56:50Z 2024-03-22T05:56:58.639 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:56:58 smithi186 ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a[66289]: logger=tracing t=2024-03-22T05:56:58.630139351Z level=info msg="Closing tracing" 2024-03-22T05:56:58.895 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:56:58 smithi186 podman[102780]: 2024-03-22 05:56:58.640062508 +0000 UTC m=+0.114723060 container died 16fd57dbbd7124c4d8f0f49a30de3b13acc12f8f5c9887ebc108931fa1c6ecc6 (image=quay.io/ceph/grafana:9.4.12, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a, maintainer=Grafana Labs ) 2024-03-22T05:56:59.294 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:56:58 smithi186 podman[102780]: 2024-03-22 05:56:58.979238596 +0000 UTC m=+0.453899148 container cleanup 16fd57dbbd7124c4d8f0f49a30de3b13acc12f8f5c9887ebc108931fa1c6ecc6 (image=quay.io/ceph/grafana:9.4.12, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a, maintainer=Grafana Labs ) 2024-03-22T05:56:59.294 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:56:58 smithi186 bash[102780]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a 2024-03-22T05:56:59.294 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:56:59 smithi186 podman[102794]: 2024-03-22 05:56:59.138707042 +0000 UTC m=+0.498085010 container remove 16fd57dbbd7124c4d8f0f49a30de3b13acc12f8f5c9887ebc108931fa1c6ecc6 (image=quay.io/ceph/grafana:9.4.12, name=ceph-f59f36e2-e80a-11ee-95cd-87774f69a715-grafana-a, maintainer=Grafana Labs ) 2024-03-22T05:56:59.749 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:56:59 smithi186 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@grafana.a.service: Deactivated successfully. 2024-03-22T05:56:59.750 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:56:59 smithi186 systemd[1]: Stopped Ceph grafana.a for f59f36e2-e80a-11ee-95cd-87774f69a715. 2024-03-22T05:56:59.750 INFO:journalctl@ceph.grafana.a.smithi186.stdout:Mar 22 05:56:59 smithi186 systemd[1]: ceph-f59f36e2-e80a-11ee-95cd-87774f69a715@grafana.a.service: Consumed 11.719s CPU time. 2024-03-22T05:57:10.977 DEBUG:teuthology.orchestra.run.smithi005:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-22T05:57:11.014 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-22T05:57:11.043 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-22T05:57:11.045 DEBUG:teuthology.misc:Transferring archived files from smithi005:/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/crash to /home/teuthworker/archive/phlogistonjohn-2024-03-22_03:02:08-orch:cephadm-wip-phlogistonjohn-testing-2024-03-21-1835-distro-default-smithi/7616054/remote/smithi005/crash 2024-03-22T05:57:11.046 DEBUG:teuthology.orchestra.run.smithi005:> sudo tar c -f - -C /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/crash -- . 2024-03-22T05:57:11.094 INFO:teuthology.orchestra.run.smithi005.stderr:tar: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/crash: Cannot open: No such file or directory 2024-03-22T05:57:11.094 INFO:teuthology.orchestra.run.smithi005.stderr:tar: Error is not recoverable: exiting now 2024-03-22T05:57:11.096 DEBUG:teuthology.misc:Transferring archived files from smithi186:/var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/crash to /home/teuthworker/archive/phlogistonjohn-2024-03-22_03:02:08-orch:cephadm-wip-phlogistonjohn-testing-2024-03-21-1835-distro-default-smithi/7616054/remote/smithi186/crash 2024-03-22T05:57:11.097 DEBUG:teuthology.orchestra.run.smithi186:> sudo tar c -f - -C /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/crash -- . 2024-03-22T05:57:11.129 INFO:teuthology.orchestra.run.smithi186.stderr:tar: /var/lib/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/crash: Cannot open: No such file or directory 2024-03-22T05:57:11.130 INFO:teuthology.orchestra.run.smithi186.stderr:tar: Error is not recoverable: exiting now 2024-03-22T05:57:11.131 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-22T05:57:11.131 DEBUG:teuthology.orchestra.run.smithi005:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-22T05:57:11.165 INFO:teuthology.orchestra.run.smithi005.stdout:2024-03-22T05:15:55.622597+0000 mon.a (mon.0) 166 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-22T05:57:11.167 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-22T05:57:11.167 DEBUG:teuthology.orchestra.run.smithi005:> sudo egrep '\[SEC\]' /var/log/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-22T05:57:11.237 DEBUG:teuthology.orchestra.run.smithi005:> sudo egrep '\[ERR\]' /var/log/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-22T05:57:11.307 DEBUG:teuthology.orchestra.run.smithi005:> sudo egrep '\[WRN\]' /var/log/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-22T05:57:11.375 INFO:teuthology.orchestra.run.smithi005.stdout:2024-03-22T05:15:55.622597+0000 mon.a (mon.0) 166 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-22T05:57:11.376 INFO:tasks.cephadm:Compressing logs... 2024-03-22T05:57:11.377 DEBUG:teuthology.orchestra.run.smithi005:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-22T05:57:11.419 DEBUG:teuthology.orchestra.run.smithi186:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-22T05:57:11.445 INFO:teuthology.orchestra.run.smithi005.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-22T05:57:11.445 INFO:teuthology.orchestra.run.smithi005.stderr:: No such file or directory 2024-03-22T05:57:11.445 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/ceph-mon.a.log 2024-03-22T05:57:11.445 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/ceph.audit.log 2024-03-22T05:57:11.445 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/ceph-osd.1.log 2024-03-22T05:57:11.445 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/ceph-client.rgw.foo.smithi005.exdhqu.log 2024-03-22T05:57:11.445 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/ceph-osd.3.log 2024-03-22T05:57:11.450 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/ceph-client.rgw.foo.smithi005.exdhqu.log: gzip -5 --verbose -- /var/log/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/ceph-mgr.y.log 2024-03-22T05:57:11.450 INFO:teuthology.orchestra.run.smithi005.stderr: 76.1% -- replaced with /var/log/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/ceph-client.rgw.foo.smithi005.exdhqu.log.gz 2024-03-22T05:57:11.450 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/tcmu-runner.log 2024-03-22T05:57:11.450 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/ceph-osd.0.log 2024-03-22T05:57:11.450 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/tcmu-runner.log: gzip -5 --verbose -- /var/log/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/ceph.cephadm.log 2024-03-22T05:57:11.450 INFO:teuthology.orchestra.run.smithi005.stderr: 88.4% -- replaced with /var/log/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/tcmu-runner.log.gz 2024-03-22T05:57:11.452 INFO:teuthology.orchestra.run.smithi186.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-22T05:57:11.452 INFO:teuthology.orchestra.run.smithi186.stderr:: No such file or directory 2024-03-22T05:57:11.452 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/ceph-osd.6.log 2024-03-22T05:57:11.452 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/ceph.audit.log 2024-03-22T05:57:11.452 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/ceph-osd.4.log 2024-03-22T05:57:11.452 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/ceph-osd.6.log: /var/log/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/ceph-osd.7.log 2024-03-22T05:57:11.452 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/ceph-osd.5.log 2024-03-22T05:57:11.453 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/ceph-osd.4.log: /var/log/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/ceph-mgr.x.log 2024-03-22T05:57:11.453 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/ceph-client.rgw.foo.smithi186.mcgodk.log 2024-03-22T05:57:11.453 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/ceph.cephadm.log 2024-03-22T05:57:11.453 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/ceph-client.rgw.foo.smithi186.mcgodk.log: 76.4% -- replaced with /var/log/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/ceph-client.rgw.foo.smithi186.mcgodk.log.gz 2024-03-22T05:57:11.453 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/ceph.log 2024-03-22T05:57:11.455 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/ceph.cephadm.log: 91.4% -- replaced with /var/log/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/ceph.audit.log.gz 2024-03-22T05:57:11.455 INFO:teuthology.orchestra.run.smithi186.stderr: 96.0% -- replaced with /var/log/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/ceph.cephadm.log.gz 2024-03-22T05:57:11.455 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/ceph-volume.log 2024-03-22T05:57:11.455 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/ceph-mon.b.log 2024-03-22T05:57:11.456 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/ceph-osd.0.log: 94.6% -- replaced with /var/log/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/ceph.audit.log.gz 2024-03-22T05:57:11.459 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/ceph-volume.log: 89.0% -- replaced with /var/log/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/ceph.log.gz 2024-03-22T05:57:11.460 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/ceph.log 2024-03-22T05:57:11.460 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/ceph-osd.2.log 2024-03-22T05:57:11.462 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/ceph.log: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-22T05:57:11.462 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/ceph-volume.log 2024-03-22T05:57:11.467 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/ceph-osd.2.log: 93.8% 97.1% -- replaced with /var/log/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/ceph.log.gz -- replaced with /var/log/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/ceph.cephadm.log.gz 2024-03-22T05:57:11.468 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-03-22T05:57:11.468 INFO:teuthology.orchestra.run.smithi005.stderr:gzip -5 --verbose -- /var/log/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/ceph-mon.c.log 2024-03-22T05:57:11.472 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/ceph-mon.b.log: 91.1% -- replaced with /var/log/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/ceph-volume.log.gz 2024-03-22T05:57:11.480 INFO:teuthology.orchestra.run.smithi186.stderr: 90.8% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-22T05:57:11.484 INFO:teuthology.orchestra.run.smithi005.stderr:/var/log/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/ceph-volume.log: /var/log/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/ceph-mon.c.log: 91.1% -- replaced with /var/log/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/ceph-volume.log.gz 2024-03-22T05:57:11.605 INFO:teuthology.orchestra.run.smithi186.stderr: 89.8% -- replaced with /var/log/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/ceph-mgr.x.log.gz 2024-03-22T05:57:11.988 INFO:teuthology.orchestra.run.smithi005.stderr: 89.0% -- replaced with /var/log/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/ceph-mgr.y.log.gz 2024-03-22T05:57:12.042 INFO:teuthology.orchestra.run.smithi005.stderr: 92.4% -- replaced with /var/log/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/ceph-mon.c.log.gz 2024-03-22T05:57:12.112 INFO:teuthology.orchestra.run.smithi186.stderr: 92.3% -- replaced with /var/log/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/ceph-mon.b.log.gz 2024-03-22T05:57:12.844 INFO:teuthology.orchestra.run.smithi005.stderr: 93.3% -- replaced with /var/log/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/ceph-osd.2.log.gz 2024-03-22T05:57:13.176 INFO:teuthology.orchestra.run.smithi186.stderr: 93.2% -- replaced with /var/log/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/ceph-osd.6.log.gz 2024-03-22T05:57:13.362 INFO:teuthology.orchestra.run.smithi186.stderr: 93.8% -- replaced with /var/log/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/ceph-osd.7.log.gz 2024-03-22T05:57:13.385 INFO:teuthology.orchestra.run.smithi005.stderr: 91.0% -- replaced with /var/log/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/ceph-mon.a.log.gz 2024-03-22T05:57:13.550 INFO:teuthology.orchestra.run.smithi186.stderr: 93.4% -- replaced with /var/log/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/ceph-osd.5.log.gz 2024-03-22T05:57:13.619 INFO:teuthology.orchestra.run.smithi005.stderr: 93.5% -- replaced with /var/log/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/ceph-osd.1.log.gz 2024-03-22T05:57:13.672 INFO:teuthology.orchestra.run.smithi186.stderr: 93.5% -- replaced with /var/log/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/ceph-osd.4.log.gz 2024-03-22T05:57:13.674 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-03-22T05:57:13.674 INFO:teuthology.orchestra.run.smithi186.stderr:real 0m2.240s 2024-03-22T05:57:13.674 INFO:teuthology.orchestra.run.smithi186.stderr:user 0m8.516s 2024-03-22T05:57:13.674 INFO:teuthology.orchestra.run.smithi186.stderr:sys 0m0.346s 2024-03-22T05:57:13.728 INFO:teuthology.orchestra.run.smithi005.stderr: 93.5% -- replaced with /var/log/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/ceph-osd.0.log.gz 2024-03-22T05:57:13.977 INFO:teuthology.orchestra.run.smithi005.stderr: 93.4% -- replaced with /var/log/ceph/f59f36e2-e80a-11ee-95cd-87774f69a715/ceph-osd.3.log.gz 2024-03-22T05:57:13.979 INFO:teuthology.orchestra.run.smithi005.stderr: 2024-03-22T05:57:13.979 INFO:teuthology.orchestra.run.smithi005.stderr:real 0m2.548s 2024-03-22T05:57:13.979 INFO:teuthology.orchestra.run.smithi005.stderr:user 0m11.095s 2024-03-22T05:57:13.979 INFO:teuthology.orchestra.run.smithi005.stderr:sys 0m0.404s 2024-03-22T05:57:13.979 INFO:tasks.cephadm:Archiving logs... 2024-03-22T05:57:13.980 DEBUG:teuthology.misc:Transferring archived files from smithi005:/var/log/ceph to /home/teuthworker/archive/phlogistonjohn-2024-03-22_03:02:08-orch:cephadm-wip-phlogistonjohn-testing-2024-03-21-1835-distro-default-smithi/7616054/remote/smithi005/log 2024-03-22T05:57:13.981 DEBUG:teuthology.orchestra.run.smithi005:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-22T05:57:15.518 DEBUG:teuthology.misc:Transferring archived files from smithi186:/var/log/ceph to /home/teuthworker/archive/phlogistonjohn-2024-03-22_03:02:08-orch:cephadm-wip-phlogistonjohn-testing-2024-03-21-1835-distro-default-smithi/7616054/remote/smithi186/log 2024-03-22T05:57:15.519 DEBUG:teuthology.orchestra.run.smithi186:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-22T05:57:16.799 INFO:tasks.cephadm:Removing cluster... 2024-03-22T05:57:16.799 DEBUG:teuthology.orchestra.run.smithi005:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 --force 2024-03-22T05:57:17.098 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid f59f36e2-e80a-11ee-95cd-87774f69a715 --force 2024-03-22T05:57:17.392 INFO:tasks.cephadm:Removing cephadm ... 2024-03-22T05:57:17.392 DEBUG:teuthology.orchestra.run.smithi005:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-22T05:57:17.407 DEBUG:teuthology.orchestra.run.smithi186:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-22T05:57:17.421 INFO:tasks.cephadm:Teardown complete 2024-03-22T05:57:17.421 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-22T05:57:17.432 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-22T05:57:17.432 DEBUG:teuthology.orchestra.run.smithi005:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-22T05:57:17.449 DEBUG:teuthology.orchestra.run.smithi186:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-22T05:57:17.460 INFO:teuthology.orchestra.run.smithi005.stderr:bash: line 1: ntpq: command not found 2024-03-22T05:57:17.464 INFO:teuthology.orchestra.run.smithi005.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-22T05:57:17.464 INFO:teuthology.orchestra.run.smithi005.stdout:=============================================================================== 2024-03-22T05:57:17.464 INFO:teuthology.orchestra.run.smithi005.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 20 +67us[ +71us] +/- 78ms 2024-03-22T05:57:17.464 INFO:teuthology.orchestra.run.smithi005.stdout:^* hv02.front.sepia.ceph.com 3 6 377 19 -954us[ -950us] +/- 69ms 2024-03-22T05:57:17.465 INFO:teuthology.orchestra.run.smithi005.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 21 +995us[ +999us] +/- 67ms 2024-03-22T05:57:17.465 INFO:teuthology.orchestra.run.smithi005.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-22T05:57:17.474 INFO:teuthology.orchestra.run.smithi186.stderr:bash: line 1: ntpq: command not found 2024-03-22T05:57:17.478 INFO:teuthology.orchestra.run.smithi186.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-22T05:57:17.478 INFO:teuthology.orchestra.run.smithi186.stdout:=============================================================================== 2024-03-22T05:57:17.478 INFO:teuthology.orchestra.run.smithi186.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 18 +63us[ +63us] +/- 78ms 2024-03-22T05:57:17.478 INFO:teuthology.orchestra.run.smithi186.stdout:^* hv02.front.sepia.ceph.com 3 6 377 21 -985us[ -988us] +/- 69ms 2024-03-22T05:57:17.478 INFO:teuthology.orchestra.run.smithi186.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 18 +981us[ +981us] +/- 67ms 2024-03-22T05:57:17.478 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-22T05:57:17.479 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-22T05:57:17.488 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-22T05:57:17.489 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-22T05:57:17.499 DEBUG:teuthology.orchestra.run.smithi005:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-22T05:57:17.600 DEBUG:teuthology.orchestra.run.smithi186:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-22T05:57:17.681 DEBUG:teuthology.orchestra.run.smithi005:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-22T05:57:17.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-22T05:57:17.712 DEBUG:teuthology.orchestra.run.smithi186:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-22T05:57:17.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-22T05:57:17.743 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-22T05:57:17.752 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-22T05:57:17.761 INFO:teuthology.task.internal:Duration was 2870.069081 seconds 2024-03-22T05:57:17.761 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-22T05:57:17.770 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-22T05:57:17.770 DEBUG:teuthology.orchestra.run.smithi005:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-22T05:57:17.773 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-22T05:57:17.812 INFO:teuthology.orchestra.run.smithi005.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-22T05:57:17.826 INFO:teuthology.orchestra.run.smithi186.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-22T05:57:18.241 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-22T05:57:18.242 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi005.front.sepia.ceph.com 2024-03-22T05:57:18.242 DEBUG:teuthology.orchestra.run.smithi005:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-22T05:57:18.301 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi186.front.sepia.ceph.com 2024-03-22T05:57:18.301 DEBUG:teuthology.orchestra.run.smithi186:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-22T05:57:18.328 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-22T05:57:18.328 DEBUG:teuthology.orchestra.run.smithi005:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-22T05:57:18.347 DEBUG:teuthology.orchestra.run.smithi186:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-22T05:57:18.401 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-22T05:57:18.401 DEBUG:teuthology.orchestra.run.smithi005:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-22T05:57:18.412 DEBUG:teuthology.orchestra.run.smithi186:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-22T05:57:19.080 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-22T05:57:19.090 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-22T05:57:19.091 DEBUG:teuthology.orchestra.run.smithi005:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-22T05:57:19.118 DEBUG:teuthology.orchestra.run.smithi186:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-22T05:57:19.146 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-22T05:57:19.156 DEBUG:teuthology.orchestra.run.smithi005:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-22T05:57:19.159 DEBUG:teuthology.orchestra.run.smithi186:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-22T05:57:19.182 INFO:teuthology.orchestra.run.smithi005.stdout:kernel.core_pattern = core 2024-03-22T05:57:19.212 INFO:teuthology.orchestra.run.smithi186.stdout:kernel.core_pattern = core 2024-03-22T05:57:19.243 DEBUG:teuthology.orchestra.run.smithi005:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-22T05:57:19.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-22T05:57:19.261 DEBUG:teuthology.orchestra.run.smithi186:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-22T05:57:19.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-22T05:57:19.297 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-22T05:57:19.308 INFO:teuthology.task.internal:Transferring archived files... 2024-03-22T05:57:19.310 DEBUG:teuthology.misc:Transferring archived files from smithi005:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/phlogistonjohn-2024-03-22_03:02:08-orch:cephadm-wip-phlogistonjohn-testing-2024-03-21-1835-distro-default-smithi/7616054/remote/smithi005 2024-03-22T05:57:19.310 DEBUG:teuthology.orchestra.run.smithi005:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-22T05:57:19.366 DEBUG:teuthology.misc:Transferring archived files from smithi186:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/phlogistonjohn-2024-03-22_03:02:08-orch:cephadm-wip-phlogistonjohn-testing-2024-03-21-1835-distro-default-smithi/7616054/remote/smithi186 2024-03-22T05:57:19.366 DEBUG:teuthology.orchestra.run.smithi186:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-22T05:57:19.411 INFO:teuthology.task.internal:Removing archive directory... 2024-03-22T05:57:19.411 DEBUG:teuthology.orchestra.run.smithi005:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-22T05:57:19.413 DEBUG:teuthology.orchestra.run.smithi186:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-22T05:57:19.458 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-22T05:57:19.470 INFO:teuthology.task.internal:Not uploading archives. 2024-03-22T05:57:19.470 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-22T05:57:19.480 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-22T05:57:19.480 DEBUG:teuthology.orchestra.run.smithi005:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-22T05:57:19.482 DEBUG:teuthology.orchestra.run.smithi186:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-22T05:57:19.494 INFO:teuthology.orchestra.run.smithi005.stdout: 265305 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 22 05:57 /home/ubuntu/cephtest 2024-03-22T05:57:19.514 INFO:teuthology.orchestra.run.smithi186.stdout: 265305 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 22 05:57 /home/ubuntu/cephtest 2024-03-22T05:57:19.514 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-22T05:57:19.525 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-22T05:57:19.546 INFO:teuthology.nuke:Checking targets against current locks 2024-03-22T05:57:19.575 DEBUG:teuthology.nuke:shortname: smithi005 2024-03-22T05:57:19.575 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-22T05:57:19.595 DEBUG:teuthology.nuke:shortname: smithi186 2024-03-22T05:57:19.595 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-22T05:57:19.599 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi005.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/phlogistonjohn-2024-03-22_03:02:08-orch:cephadm-wip-phlogistonjohn-testing-2024-03-21-1835-distro-default-smithi/7616054', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-22 05:00:46.649248', 'locked_by': 'scheduled_phlogistonjohn@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPzVmRysK6cWm/K7TyDn26pLTrKjNjf5YsJF69j9P2iKhMN0Tsq0RxTMyRjh9Q7SWLM4d2lGNxhhCaLJX01mwUE='} 2024-03-22T05:57:19.616 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi186.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/phlogistonjohn-2024-03-22_03:02:08-orch:cephadm-wip-phlogistonjohn-testing-2024-03-21-1835-distro-default-smithi/7616054', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-22 05:00:46.647484', 'locked_by': 'scheduled_phlogistonjohn@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD/uivWwKGQpDzapTJ7C4+YQ5E/K0dCT4rdP4t5E2UfJKmnfA1EA5gLKYjQDFUneT0wUQfiUXdVbs4MogdIBAFg='} 2024-03-22T05:57:19.644 INFO:teuthology.orchestra.console.smithi005:Power off 2024-03-22T05:57:19.645 DEBUG:teuthology.orchestra.console.smithi005:pexpect command: ipmitool -H smithi005.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-22T05:57:19.669 DEBUG:teuthology.orchestra.console.smithi005:power off output: Chassis Power Control: Down/Off 2024-03-22T05:57:19.669 DEBUG:teuthology.orchestra.console.smithi005:pexpect command: ipmitool -H smithi005.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-22T05:57:19.685 INFO:teuthology.orchestra.console.smithi186:Power off 2024-03-22T05:57:19.685 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-22T05:57:19.694 DEBUG:teuthology.orchestra.console.smithi005:check power output: Chassis Power is on 2024-03-22T05:57:19.707 DEBUG:teuthology.orchestra.console.smithi186:power off output: Chassis Power Control: Down/Off 2024-03-22T05:57:19.707 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-22T05:57:19.734 DEBUG:teuthology.orchestra.console.smithi186:check power output: Chassis Power is on 2024-03-22T05:57:23.695 DEBUG:teuthology.orchestra.console.smithi005:pexpect command: ipmitool -H smithi005.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-22T05:57:23.735 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-22T05:57:23.807 DEBUG:teuthology.orchestra.console.smithi005:check power output: Chassis Power is on 2024-03-22T05:57:23.846 DEBUG:teuthology.orchestra.console.smithi186:check power output: Chassis Power is on 2024-03-22T05:57:27.811 DEBUG:teuthology.orchestra.console.smithi005:pexpect command: ipmitool -H smithi005.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-22T05:57:27.847 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-22T05:57:27.923 DEBUG:teuthology.orchestra.console.smithi005:check power output: Chassis Power is off 2024-03-22T05:57:27.958 DEBUG:teuthology.orchestra.console.smithi186:check power output: Chassis Power is off 2024-03-22T05:57:28.024 INFO:teuthology.orchestra.console.smithi005:Power off completed 2024-03-22T05:57:28.059 INFO:teuthology.orchestra.console.smithi186:Power off completed 2024-03-22T05:57:28.163 INFO:teuthology.run:Summary data: description: orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/classic} duration: 2870.069081068039 failure_reason: '"2024-03-22T05:15:55.622597+0000 mon.a (mon.0) 166 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log' owner: scheduled_phlogistonjohn@teuthology success: false 2024-03-22T05:57:28.163 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-22T05:57:28.243 INFO:teuthology.run:FAIL